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ABSTRACT 

This document contains eight papers from an action 
research program to foster good practice in adult literacy provision 
and policy. "Introduction" (J. Joy Cumming, Christina E. van 
Kraayenoord) presents an overview of the action research project and 
individual reports. "Assessment: Making a Difference in Adult 
Literacy and Numeracy Learning" (J. Joy Cumming, Christina E. van 
Kraayenoord) summarizes the assessment issues presented at the 
project seminars. The following six papers present the outcomes of 
the individual research projects: "Assessment as an Ongoing Feature 
of the Learning Environment" (Catherine Doherty, Francis Mangubhai, 
Joan Shearer); "Filling in the Blanks: Construct Validity Issues in 
the Development of Assessment Tasks for Adult Literacy Students" (Ann 
Kelly); "Moderation of Competency-Based Assessment: Issues and 
Practice in Adult Literacy and Numeracy" (John Bailey et al.); "Let's 
Start Reading: The Field Trial of a Computer Program as an Aid in 
Adult Literacy Teaching" (Maureen Mortimer); "Who's Learning from 
Whom: A New Teacher Explores Literacy Assessment and Teaching 
Strategies in an Adult Urban Classroom" (Cherie de Pinna) ; and "Are 
Literacy and Numeracy Competencies Achieved by Newstart Clients 
Transferred to the Clients' Real-Life Context?" (Maureen Cooper, 

Susan Garside). The final paper, "Writing an Action Research 
Proposal" (Peter Mountney) describes the focus of the program's 
research proposal workshops. Contains 12 references. (MN) 
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INTRODUCTION 

J. Joy Cumming 

Christina E. van Kraayenoord 



In 1995, the National Languages and Literacy Institute of Australia (NLLLA) Adult Literacy 
Research Network (Queensland) undertook to support a program of action research. This activity 
was part of an agenda developed by the Network at the national level to foster the development 
of good practice in adult literacy provision and policy through an amalgamation of the efforts of 
academics, teachers and policy makers. The Queensland Network statement of aims and objec- 
tives identifies the major roles of the Network to be to support the dissemination of research into 
practice and to facilitate of procedures to link research and practice. Action research involving 
collaboration between researchers, teachers as researchers and others is becoming recognised as 
the most effective means to address areas of significant need identified in and by the ‘field’. 

A structure was developed to support such involvement in research. Firstly, the Adult Literacy 
Research Network Steering Committee (Queensland) noted that the area of assessment was 
becoming ever more important in the adult literacy and numeracy sector. Changes such as pro- 
gram accountability, the introduction of accredited curriculum in adult basic education and sys- 
tems such as the National Reporting System require significant and demanding attention from 
practitioners and policy makers. The theme for the action research projects was thus set to be 
‘Assessment’. 

A series of professional development activities was organised throughout Queensland. 
Seminar/workshops on the general theme of assessment were held in four locations in 
Queensland in April and May 1995. Christa van Kraayenoord of The University of Queensland 
led sessions in Townsville and Cairns, while Joy Cumming of Griffith University led sessions in 
Rockhampton and Brisbane. The seminars were free and open to any who wished to attend with 
corresponding good attendance. Many adult literacy practitioners travelled long distances at their 
own costs and in their own time to attend the sessions, showing how much such professionals 
value opportunities for professional development. Workshops on the writing of a research pro- 
posal were led by Peter Mountney of Griffith University at the same four sites. Again these were 
open for all to attend, although there was an expectation that those attending would have a 
research topic they wished to investigate through an action research project. 

Following the seminars, proposals were sought from groups or individuals. The Network was 
pleased that in only the second year of a program of action research support, six proposals of 
high quality were submitted. All six were able to be funded. The amount of funding is necessari- 
ly small and quality for money has been provided in return. At this point, we would like to 
acknowledge and thank the contributions and efforts of all of those who have participated in 
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these projects, whether as researchers, teachers or participants in the courses. The adult lit- 
eracy and numeracy field has always been noted for its professionalism and dedication and 
despite a changing environment, this shows no signs of abatement. 

The outcome of the action research program is presented in the following chapters. The first 
chapter provides a summary of the issues in assessment that were presented at our seminars. 
We have emphasised both in the chapter and in the seminars general assessment issues and 
theoretical underpinnings that all educators need to address. The issues discussed are not 
restricted to adult literacy and numeracy provision, to the vocational sector, or to competen- 
cy-based assessment. Good assessment practice in any context requires the recognition of 
and adherence to the same fundamental concerns about appropriateness, validity and 
responsiveness to the learning goals of instruction. 

The next six chapters present the outcomes of the individual research projects. In presenting 
these projects only very light editing has been undertaken. We hope that the value of the 
reports presented in their original formats is threefold. Firstly, the findings of the research 
studies are in themselves significant. All adult literacy practitioners will identify with some 
of the situations and issues discussed and resolved. Thus the studies are worthy of dissemi- 
nation in their own right as research in the area of adult literacy and numeracy. 

The second value of the reports is to demonstrate that educational research can be undertak- 
en in many different ways. Today there is no one research approach that is more valued 
than another. What is most important is that the persons undertaking the research perceive 
an issue that is of significance to them and explore that issue in a systematic way. 

The third value of the report chapters is the demonstration that research can also be report- 
ed in many different ways. The variety of formats, styles and orientations of the action 
research teams have been maintained deliberately. Different referencing systems are used. 
Again, this has been done to indicate clearly that it is the substance and intention of 
research that gives it quality. There is no one correct procedure or format that must be fol- 
lowed rigidly. How often are we afraid to undertake new activities (leaving aside issues of 
work and life demands) for fear of not knowing the ‘right’ way to proceed. And does this 
hesitation not resonate with the fears of so many adult literacy and numeracy students that 
they might not get it ‘right’? The research papers that follow are all excellent in their own 
right. They address issues and concerns of importance to the research teams. They site then- 
work in what is already known, if possible. They report their findings in systematic and 
well-documented ways. 

The first study, reported in Chapter 2, examines the issue of assessment from a broad per- 
spective. Catherine Doherty, Francis Mangubhai and Joan Shearer note the changing envi- 
ronment of adult literacy and numeracy provision and the new agendas of assessment and 
reporting, agendas that ‘can create real conflicts for the provider’. For their study, the 
research team defined assessment to include both formal and informal assessment and tried 
to ensure that both were captured by their research. Catherine, Francis and Joan, through 
surveys of 37 adult literacy practitioners in Queensland, examine teacher attitudes to assess- 
ment as well as their assessment practices. The study was undertaken in the context of 
agendas such as the new Queensland curriculum of the Certificate of Vocational Access 1 
(CNL03) and policy requirements such as system-wide reporting. Among their findings the 
research team note that, while the full-time and more qualified staff reported moderate 
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levels of assessment professional development, part-time teachers and coordinators reported min- 
imal to moderate levels. Professional development in assessment for these teachers, particularly if 
there is further casualisation of literacy staff, is an important consideration for the future. 

Three-quarters of respondents in the study by Catherine, Francis and Joan were using curriculum 
with specified learning outcomes against which most were required to report on students. Most 
respondents indicated the use of a considerable range of assessment techniques including infor- 
mal testing and with less emphasis on decontextualised skill testing. This reversed findings of 
previous research (Falk, 1995), and perhaps indicates that the new agendas have changed the 
emphasis and focus of assessment, as well as raising teachers’ awareness about the diverse roles 
and forms of assessment. Their findings document teachers’ reactions, both positive and negative, 
to the changes. We’re sure most practitioners, in any field, reading this chapter will empathise 
with at least some of the comments. 

The study by Catherine, Francis and Joan confirms the primacy assessment and reporting have 
gained in adult literacy and numeracy provision in a relatively short time. Respondents raise seri- 
ous issues about formalised assessment requirements during the curriculum, impacts on students 
(both positive and negative) and specific issues for assessment of students with learning difficul- 
ties and disabilities. 2 

It is unlikely that assessment requirements will now diminish. The challenge is before us all to 
maximise the quality of our practices while we seek to minimise unnecessary duplication of 
activities, on behalf of both teachers and students. 

Improving practice and the understanding of teachers’ roles in undertaking assessment are the 
focus of the next two studies in the book. Ann Kelly, in conjunction with Sharon Mullins and 
Ruth Kemp, examined moderation and construct validity issues related to assessment in adult lit- 
eracy. The framework for their study is based on the construct validity definitions emphasised by 
the work of Hagar, Athanasou and Gonczi (1994). Validity of assessment has traditionally been 
seen as having four main focuses: face, content, construct and predictive validity. Of these, con- 
struct validity, the match between an assessment activity and the performance it seeks to assess, 
has always been seen to be both the most important and most difficult. In our chapter on assess- 
ment, we mention new conceptions in validity emphasised by Messick (1989), conceptions that 
focus not only on the appropriateness of the task (that ie, traditional construct validity) but also 
the appropriateness of the interpretation and use of the outcomes. 

While Ann and her colleagues draw upon the Queensland Certificate of Vocational Access as the 
data source for the study, the issues raised about specificity and breadth of assessment tasks are 
critical and relevant to all instructional settings. They describe and use the model proposed by 
Linn, Baker and Dunbar (1991) to evaluate assessment in the Queensland Certificate. The model 
establishes eight criteria - consequences, fairness, transferability and generalisability, cognitive 
complexity, content quality, content coverage, meaningfulness and cost and efficiency - to be 
used in considering the quality of an assessment program. 

The NLLIA action research grants in Queensland allowed John Bailey, Pam Weston, Shirley 
Verrall and Jeremy Audaus, all of the Barrier Reef Institute of TAFE, to interact in the develop- 
ment and moderation of exemplar assessment materials. Such interaction is not simple when the 
bases of members of one institute are spread over hundreds of kilometres. Their chapter, Chapter 
3, describes briefly the diversity of characteristics of the adult literacy and numeracy students at 
each site. 
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John Bailey and his colleagues undertook to trial procedures for moderation of competency- 
based assessment tasks. The new Queensland curriculum (CNL03) was again the focus of 
their work, in particular the Personal focus modules at levels 1 A and IB. 3 
While the project team make a number of insightful reflections about the problems of 
matching a common competency-based curriculum to the literacy and numeracy needs and 
wants of individual adult learners, it is significant that so many of the studies in these chap- 
ters used the same Personal focus modules. 

In their chapter, the research team of the far north Queensland Institute of TAFE describes 
both the processes of their deliberations as well as the outcomes. These outcomes consist of 
both their own conceptualisations of issues in assessment for adult literacy and numeracy 
teachers as well as the exemplar materials. Their statement ‘these discussions were kept at a 
forceful yet friendly level which seemed to indicate the degree to which assessment was a 
contentious issue in literacy circles’ is just one of those that indicate how important the 
opportunity to share, discuss and debate is to the educational process and professional 
development. Such an opportunity provided through this action research project is clearly 
one of the major outcomes of the research for the team. Their chapter depicts the action 
learning and action research process discussed in Kelly’s earlier chapter. The comments and 
reflections about the demands of action research are well-recognised. 

The integrated assessment tasks that are presented in Chapter 3 provide pro forma for inno- 
vative and meaningful assessment tasks for students. Teachers can use these as a basis for 
developing tasks of their own, to suit the circumstances of their students as well as the 
demands of the curriculum. Teachers can do this with some confidence that these would 
have some shared understandings among teachers of the expectations of standards of per- 
formance implicit in such tasks. 

Overall the work by these teacher-researchers shows that in order to have a common cur- 
riculum with shared interpretations of student performance levels, teacher talk, trialing of 
tasks and moderation are necessary. As their project shows, when this occurs the outcomes 
are worthwhile for both the teachers and the students. 

Among the research studies funded through the NLLIA Network in Queensland were two 
studies that looked at innovative practices and the assessment practices that flowed from 
them. Maureen Mortimer, in her Chapter charting student interaction with a computer liter- 
acy program, displays a keen sense of innovation in learning and assessment and empathy 
with student needs. Her study shows a sense of individual need for success, confidence and 
sometimes privacy. It provides a fine example of a teacher’s self-evaluation and of action 
program evaluation. The data Maureen has collected not only provide us with a record of 
the successes of the computer program, but also are being used by Maureen to inform 
future modifications. 

The level of sophistication of Maureen’s computer program, described with complete 
understatement, amazes us. The outcomes of her work show that students in general great- 
ly enjoyed the computer work. They were motivated to spend time learning with comput- 
ers, they did learn, they did gain confidence and interact more with others, and they did set 
new and significant goals. 

While the study by Maureen was an extension of the assessment theme of the work of the 
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NLLIA Adult Literacy Research Network in Queensland in 1995 and 1996, several important 
assessment issues are embedded in her work and report. As Maureen notes: 

• feedback that is immediate and positive is the most beneficial for students, 

• success can be ensured by allowing the opportunity for students to work at their individual 
levels, 

• learning and assessment activities should be meaningful and contextualised as far as possible, 

• students enjoy academic achievement, particularly improvement against their own prior 
performance, 

• students have the capacity to reflect on their own performance and suggest better ways to 
achieve their goals, and 

• students find tangible results as important as positive reinforcement. 

In Chapter 6, Cherie de Pinna shares another aspect of self-evaluation. Again, the self-evaluation 
in this study is by the teacher, not students. Her report shows a new teacher embracing all aspects 
of literacy performance and development indicated by current critical-cultural literacy. While her 
teaching work took place with a group of indigenous adult students, the principles Cherie sought 
to enact in her teaching are seen as the basis of all good andragogy, if not all good teaching. She 
looks at student participation in and management of the learning context, the sharing of under- 
standings at pedagogical, individual and cultural levels, and the tensions, mentioned by other 
teacher-researchers, between individual needs in adult literacy and core curriculum. These ten- 
sions are even more difficult for the beginning teacher seeking to resolve seemingly incompatible 
concepts of individual competency-based assessment and holistic literacy performance. Further 
work such as the exemplars developed by John, Pam, Shirley and Jeremy in Chapter 3 should 
help bridge this gap. 

The extracts from Cherie’ s journal highlight common problems and issues encountered in adult 
literacy and numeracy provision. Not least of these is the development of a real or authentic con- 
text for outcomes that while emulating life or the workplace, are still artificial in the formal edu- 
cation setting. She also wanted to develop learner responsibility and a positive group dynamic to 
enhance learning. The brief report provided by the students and their comments show that a real- 
istic context of meetings and fund-raising was established. Overall, we thank Cherie for under- 
taking a critical self-reflection that we so often ask of others, but would prefer not to do for our- 
selves. She shares with us her successes as well as some activities and plans that did not go as 
she had expected. Cherie also justly reminds us that we learn from each other, and only by being 
willing to learn and to understand, can we reach our own goals. 

The final research study report in Chapter 7 examines adult literacy assessment in the context of 
longterm student gains. When we assess in any educational or work setting, we are making 
judgements about performance. Underlying any act of assessment is the belief that the achieve- 
ment we denote through assessment is enduring. As Maureen Cooper and Susan Garside of the 
Bayside campus of the Moreton Institute of TAFE indicate in this chapter, it is also the hope of 
teachers or assessors that the performance indicated by assessment will transfer to a situation of 
use — whether it be the abstraction of mathematical problem-solving or the specifics of work 
tasks. Maureen and Susan have explored this very issue: the match and transfer of literacy and 
numeracy competencies acquired in a formal training course to the workplace. 

Once again, as the context of the study was in a Queensland TAFE setting, the focus of the study 
was the Queensland Certificate in Vocational Access, in its 1995 accredited form. Maureen and 
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Susan provide a succinct review of issues in general literacy and numeracy education and 
transfer of skills that should be informative for all practitioners. These are complex issues 
to which we still have so few answers but clearly are at the core of all education and train- 
ing. 

Maureen and Susan followed ten students who had completed a Newstart program. Some 
were in employment, some were seeking work, or were working part-time, others were in a 
new course with a work experience component. Comprehensive data from a range of stake- 
holders, particularly from work supervisors, former teachers and Commonwealth 
Employment Service (CES) case managers, were collected. We’re sure most readers will 
find the ‘critical incident’ accounts from the participants heart-warming and enlightening, 
both in terms of the outcomes and the methodology. The participants are able to provide 
many instances of their use and transfer of the literacy and numeracy skills developed 
through the Newstart programs. Their data indicate that this success may be due, at least in 
part, to emphasis placed by the Newstart program teachers on cognitive factors such as 
repeated practice and practice in multiple contexts, as well as learning factors such as inde- 
pendence, high expectations of self and others, responsibility for learning and willingness 
‘to have a go’. These latter attributes were identified by the employers interviewed as hav- 
ing as much importance as specific skills. 

Maureen and Susan raise the issue of assessment of the affective domain as this is seen to 
be so important. Now that we are addressing so many of the difficult issues in assessment, 
the assessment of attitudes and other affective attributes could be the next focus. Again 
their study reinforces our hypothesis that many students, including adults, need not only a 
sense of achievement, but a clear indication of what they have achieved. 

Finally, Chapter 8 of this book on the action research program of the NLLIA Adult Literacy 
Research Network in Queensland presents the focus of the research proposal workshops 
conducted by Peter Mountney. Peter provides a practical guide for any who wish to consid- 
er how to design and plan a small action research project. We provide this Chapter in the 
hope that many adult literacy and numeracy teachers and policy makers will be further 
enthused to participate in such action research studies. It is only through these continued 
activities that we can identify and resolve issues that are significant for practitioners, and 
can help to develop best practice in provision through the amalgamation of strong theoreti- 
cal and empirical evidence. 
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1 One of the major changes throughout Australia over the last ten years has been the develop- 
ment in every state of accredited certificates of basic education and vocational access. The 
certificate programs now legitimate and recognise basic education for adults and allow adults 
pathways to further studies. 

2 It is worth commenting here that one goal during the development of the National Reporting 
System was to ensure that even small changes for such students can be seen as significant 
progress. 

3 Levels 1 A and 1 B indicate beginning level proficiency within the curriculum document CNL03. 
The Personal focus is derived from the influence of the National Framework of Adult English 
Language, Literacy and Numeracy Competence (ACTRAC, 1993) with identifies six aspects 
of communication or contexts through which communication takes place. These are: 
Procedural, Technical, Personal, Cooperative, Systems and Public communication. This con- 
struction of communication settings has supported the development of curriculum that identi- 
fy different areas of focus according to student needs and skills. The Personal focus modules 
at this level equate to the type of work which has always been undertaken in beginning adult 
basic education in previously informal instruction. The advantage of such a formal curriculum 
is that it therefore allows formal accreditation and articulation of all education achievement. 
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ASSESSMENT: MAKING A DIFFERENCE 
IN ADULT LITERACY AND NUMERACY 
LEARNING 

J. Joy Cumming 

Christina E. van Kraayenoord 



Introduction 

The purpose of this chapter is to explore the themes of assessment that underpin the research 
projects reported in the following chapters. These themes were introduced in and emerged 
from professional development workshops with adult literacy and numeracy practitioners in 
Queensland during 1995. Throughout these workshops, the following definition of assess- 
ment was used. 

Assessment is the process of making judgements of performance , in a number of situa- 
tions , using a variety of different techniques, over time . Assessment is a way of col- 
lecting information. Assessment is not an isolated activity, but a continuous process. 
The purposes of assessment are to inform instruction and report progress . 

This definition encompasses the diversity of ways in which assessment can be both under- 
taken and used. Assessment can be used for diagnostic, formative and summative purposes, 
as well as for providing information for reporting and evaluation. 

The definition also reflects changes that have occurred over the past decade in assessment 
theory and practice, changes explored in the following discussion. The next sections of the 
chapter examine five general assessment issues: contextual factors; reasons for changes in lit- 
eracy assessment; the effects on practice of these changes; their implications for teachers; and 
future issues. 

Contextual factors contributing to an increased focus on assessment in adult literacy 
and numeracy 

Adult literacy and numeracy provision in Australia has undergone a number of significant 
changes in recent years. Historically, adult literacy and numeracy instruction originated 
through the work of community and volunteer agencies in community centres, libraries, and 
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tutors’ and students’ homes. Instruction tended to be individual or in small groups, and was pro- 
vided to meet personal goals and needs. This work still continues today. However, International 
Literacy Year in 1990 was a turning point for the quantity and nature of adult literacy and numer- 
acy provision in Australia. Various lobby groups for adult literacy and numeracy ensured that 
International Literacy Year received wide public attention, with a number of policy and language 
statements and research initiatives ensuing. This attention, and subsequent recognition of unmet 
need, led to a considerable increase in state and federal funding and provision in adult literacy and 
numeracy. 

Coincident with these initiatives, major changes were occurring in the organisation of the 
Australian workforce. Award restructuring and enterprise agreements were introduced to facilitate 
the development of and to broaden workers’ skills. Demarcation of jobs and tasks was reduced. 
The rationale for these changes was an expectation that workers would become more skilled, and 
hence able to move between jobs more easily or to advance in any sector - so creating a ‘clever 
country’. At the same time, the nature of jobs themselves changed, with demands for higher liter- 
acy skills to cope with technological developments, increased paper work, and training itself. 
Literacy was therefore perceived, by those internal and external to the workplace, as an integral 
component of the transition process for workers. Specific vocational and literacy skills were taught 
via workplace training. Funding for such training was increased by the introduction of the Training 
Levy on large companies, as well as initiatives such as Workplace English Language and Literacy 
(WELL) funding. 

The recognition of literacy as an important factor in success in work, combined with high unem- 
ployment rates, also led to an increasing emphasis on and provision of literacy and numeracy 
instruction in labour market programs for the unemployed. Differences in provision in communi- 
ty basic education, vocational and workplace training started to blur, with the same teachers, often 
from technical and further education (TAFE) institutions, instructing in all sectors. 

The dramatic rise in provision over the past decade is directly tied to improvements in state and 
federal funding for such programs, paid from the public purse. With increases in funding came 
strong and insistent demands for accountability. In the past, information about accountability has 
been satisfied by raw and questionable figures: numbers of students beginning and completing pro- 
grams, numbers of students getting jobs or going to further training, and cost efficiency per ‘unit’. 
However, an important aspect of accountability should be for teachers to be able to demonstrate 
what students have learned as a result of their participation in training, as well as to relate these out- 
comes to program goals and content. The present shift towards accountability in terms of learning 
outcomes is not only educationally desirable, it is necessary to ensure that only high quality pro- 
grams are funded. 

The development of accredited curricula and competency-based assessments, and the ‘recognition 
of prior learning’ should assist teachers to meet the demands of accountability. These develop- 
ments create recognised pathways for workers and students, allowing clearer articulation between 
levels of certification and for employment. 

Many teachers in adult literacy and numeracy who have their origins in community provision and 
commitment to the welfare of individuals, are uncomfortable with the rapid introduction of all of 
these changes. However, recognition of the needs and individuality of the students is not neces- 
sarily lost. Students, themselves, need and want to know how they are achieving and about their 
progress through their course, as well as to have documented evidence of their learning. 
Accountability to the students, above everything else, is paramount. 
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Changes in the nature of literacy and numeracy assessment: Reasons 

Several factors have led to changes in the ways in which literacy and numeracy are being 
assessed. Firstly, our awareness of the nature of literacy learning and factors affecting liter- 
acy performance has changed considerably in recent times. No longer is literacy seen only 
as the ability to decode and to comprehend written text. More recent conceptualisations of 
literacy take into account the constructive and interactive nature of literacy learning, focus- 
ing on the text, the task, the reader and the context. Similarly, conceptions of numeracy have 
evolved from notions of arithmetic and basic computation to incorporate workplace mathe- 
matics, statistical applications and understanding, and problem-solving skills. The role of 
language in numeracy and areas such as technological, computer and quantitative literacies 
are also being considered. Hence there is now an awareness of the multiplicity of literacy, or 
more aptly, literacies. We are all more literate with some forms of written, oral and quanti- 
tative communication than with others, and in some contexts, than in others. 

Such complex conceptualisations of literacy and numeracy have necessitated a corresponding 
shift in the ways in which we assess adults in literacy and numeracy. Tests of unrelated words 
in oral reading, standardised comprehension tests, spelling tests, arithmetic sums, and so on, 
are no longer sufficient indicators of a student’s literacy and numeracy performance. While 
any of these activities can still have a specific function in an assessment program , 1 their nar- 
row focus and decontextualised nature limit their capacity to reflect what any student, partic- 
ularly an adult, can do. Assessment today focuses not just on product but also on process, not 
just on what a student can or can’t do, but why and how they do what they do. Metacognitive 
aspects such as learning how to learn, and reflection on learning are seen as important instruc- 
tional goals and hence require some systematic observation. The context of performance and 
attitudinal and motivational factors, seen to be intertwined with literacy and numeracy, also 
need exploration. 

A second major factor affecting assessment practice today relates to changes in assessment 
theory itself. Teachers are probably familiar with terminology such as standardised tests, for- 
mative and summative assessment, folios, reliability and validity, but may not realise that the 
area of assessment is a theoretical field of research in itself. Assessment theory has evolved 
from psychological theories of measurement (psychometrics) to educationally-based concep- 
tions of assessment of student achievement. Validity of assessment is always seen as the crit- 
ical factor for good practice. Psychometric test development depends on quantitative indica- 
tors of reliability and validity obtained from results for groups. These indicators are measures 
of the coherence of the test itself, rather than rather than focusing on the appropriateness of 
the proposed assessment relevant to the proposed learning goals of the individual. More 
importantly most psychometric tests are designed to measure theoretical attributes and under- 
lying abilities. However, the validity of the test for actually ‘measuring’ an attribute is rarely 
established beyond face validity, the items on the test ‘look’ appropriate. 

In contrast, assessment of student achievement has become more concerned with the direct 
observation of skills, knowledge and attitudes demonstrated during a course of instruction 
and in assessment situations, that is, performance-based assessment. Performance assessment 
in literacy and numeracy involves observing the behaviours and motivations of students, and 
making judgements about the work they produce in response to various written, visual and 
spoken stimuli. Today, valid assessment is no longer established by ‘good’ psychometric 
indicators, but by assessment appropriate to the context, the outcomes of which are interpreted 
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and used appropriately (Messick, 1989). 

Using this definition of validity, we can see that the best literacy and numeracy assessment prac- 
tices are going to be those that are the most appropriate assessment activities for the learning con- 
text and purpose. We noted in the Introduction that assessment serves many purposes. A single 
assessment activity may not be suitable for multiple purposes. Formative assessment is used to pro- 
vide feedback to both teacher and student about learning progress. The examination of students’ 
processes when involved in literacy and numeracy activities can be as important as observing the 
outcomes of the activities. Such information is immediately useful for the modification of teaching 
and for the student to be able to set future learning goals. 

Summative assessment can involve the observation and rating of performance on an assessment 
task or activity, as well as the aggregation of assessment outcomes to provide information for cer- 
tification. Summative assessment serves as an indication of performance standards achieved. In 
most contexts, summative assessment will report on the quality and complexity of literacy and 
numeracy produced, such as writing a report, or providing a logical solution for a mathematical 
activity. There is, however, scope to incorporate information on strategic outcomes as well, such as 
development of a strategy using context cues in reading rather than sounding out a word, or esti- 
mating reasonableness of answers in mathematical tasks. Thus, adult literacy and numeracy prac- 
titioners should see the collection and use of both formative and summative information about stu- 
dents as important. 



Changes in literacy and numeracy assessment: Effect on practice 

Changes in conceptions of literacy and numeracy and in assessment have necessarily affected the 
practice of teachers in the adult literacy and numeracy field. Performance can change according to 
context, and therefore literacy and numeracy have to be assessed across a range of activities. Prior 
knowledge, contexts of performance under speed or stress conditions, work demands and appro- 
priateness of a task can all affect performance at any time. 

Literacy is seen to involve at the very least the four macro skills of reading, writing, listening and 
speaking, as well as critical thinking across all of these. Viewing 2 has also recently appeared in 
some definitions of literacy. Comprehensive literacy assessment must incorporate all of these 
areas. Within each of these areas, performance is also complex. For example, Freebody and Luke 
(1990) have identified four components of reading performance related to reader roles as: text 
decoders, text participants, text users and text analysts. These authors see all of these components 
as essential for fluent ‘reading’, while none on its own is sufficient. In the past, teachers may have 
emphasised only one or two roles such as text decoder or text participant in both instruction and 
assessment. Adult literacy and numeracy practioners need to ensure that readers become compe- 
tent in all of these roles. Work in numeracy is following similar directions (Johnston, 1994). The 
ability of adults to act as critical users of mathematics is seen to be as important as the capacity to 
compute. The development of appropriate assessment procedures to examine these roles in litera- 
cy and numeracy is the next important step. 

Writing is no less complex an activity than reading. However, teachers often limit their comments 
on students’ proficiency in writing to consideration of the quantity and the surface lexical-syntac- 
tic features of the written text. Instead, teachers need to consider a range of expectations related to 
writing, including the use of the appropriate genre, and taking into account the audience, and 
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achievement of the purpose. These expectations should be made clear to the students. 

Speaking and listening should receive instructional and assessment attention. To date there 
has been little research on appropriate assessment approaches for adults in these domains. 
However, we believe that students should be assessed to identify how they interact in a vari- 
ety of situations with a ranges of audiences and purposes. Students’ abilities to control lin- 
guistic structures and features of spoken language, and their abilities to listen to, monitor, and 
critically reflect on their own and others communication should also be examined. 

Assessment in numeracy needs to focus on newer conceptions of numeracy and mathematics, 
including problem-solving and thinking skills. The development and assessment of the latter 
skills should be promoted, with a reduced emphasis on learning of arithmetic and computa- 
tional skills. Just as literacy performance is influenced by context, so too are mathematical 
behaviours. Research has shown that we approach and construct apparently similar mathe- 
matical tasks in different ways in different contexts. Performance is usually enhanced in con- 
text. Therefore instructional and assessment practices need to be clearly related to the appro- 
priate contexts, for example, a specific workplace, so that the situational demands can be 
taken into consideration. 

Consideration of good assessment practice encompasses many issues. Firstly, the first 
Assessment Principle of the National Reporting System (NRS) (Coates et al., 1995) recom- 
mends that as wide a range of tasks as possible be used. Such tasks can include open-ended 
formats, where students structure and present information, as well as closed-formats, where 
students are completing a more restricted activity. Assessment can be undertaken through 
both written and oral activities. Often adult literacy and numeracy students can display 
knowledge and skills more easily in an oral than in a written format. Conversely, students 
from a non-English speaking background can find written presentations more enabling of per- 
formance than oral presentations. 

Secondly, conditions of assessment will also differ. The time and resources available for an 
activity can be varied, on a continuum from limited to unspecified, to suit the instructional 
and assessment goals. Students with special needs and with English as a second language 
may require extra time to complete their work. Task and text can be modified to match the 
abilities or characteristics of students. Assessment can be embedded within instructional 
activities or completed on conclusion of instruction. In ideal situations, assessment and 
instruction are so integrated that it is not possible to distinguish between instructional and 
assessment activities. 

Most importantly, assessment is not a one-off activity, but should be based on multiple 
assessment activities through continuous assessment or approaches such as folio or portfolio 
assessment. A folio reflects a student’s performance and progress over time. It is construct- 
ed from different types of evidence representative of the curriculum goals and objectives. 
The contents of a folio should be systematically organised, so that changes in learning can be 
tracked. Folios should also be reviewed frequently so that students become conscious of what 
they have achieved, and can develop new learning goals. 

Does this mean that all performance based assessment activities are good assessment and that 
all assessment activities should be collated in a folio? No. Assessment is not a random activ- 
ity. It has to be ongoing and systematic, matching key curriculum goals and assessment 
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purposes. A teacher cannot assess everything, and therefore must choose to assess essential dimen- 
sions of literacy and numeracy performance in a course. Choices are also made when using folio 
assessment. Some exemplary folio practices allow students to select their pieces of work in col- 
laboration with their teachers. These pieces are chosen to suit the purpose of the folio. It may be: 
to show latest and best work; to show diversity through the inclusion of many different pieces of 
work: or to show the development that has occurred through the inclusion of early as well as later 
work. 

Adult literacy and numeracy teachers are often unhappy about making judgements about students’ 
work, for fear of appearing critical or affecting student self-esteem. However, performance can be 
judged in several ways: against self, against others and against specified criteria. Adult literacy and 
numeracy teachers frequently negotiate goals and curriculum content with adult learners. Working 
from these goals, the students, with the teacher, can chart their progress against their own goals. 
An advantage of this approach is not only that the student is able to control and have ownership of 
learning, but also that the reflective act of self-assessment should enhance learning. 

Judgement against the performance of others is usually referred to as normative assessment. While 
this may have negative connotations for many teachers and students, situations can occur where it 
is appropriate. At times, such as for diagnostic purposes, it may be beneficial to know that a stu- 
dent’s performance on a common task is equivalent to that of only a small percentage of the pop- 
ulation and highly atypical. Normative or ranking decisions are also necessary for entry to courses 
and institutions such as universities or the awarding of scholarships. Normative comparisons are 
simple (deceptively) when based on a percentage, a score or a single letter. However these com- 
parisons are more difficult when based on complex assessment outcomes such as folios and aggre- 
gated outcomes. While there are problems to be addressed, folio assessments have been used suc- 
cessfully to make normative comparisons. 

Criterion-referenced assessment where performance is judged against explicit criteria has grown in 
popularity. In this approach, performance is judged against criteria that are made explicit. In the 
best implementations of criterion-referenced assessment, standards for the criteria are also provid- 
ed (Sadler, 1987). The benefits of criterion-referenced assessment are not only that the goals and 
expectations are known to the students, but also that all students can achieve at the highest level of 
performance. Knowledge of the criteria and standards allows students to self-monitor their perfor- 
mance. 

Competency-based assessment can be regarded as a specific form of criterion-referenced assess- 
ment. It involves the identification of competencies that the student is stated to have achieved or 
have not yet achieved. In some instances, progress towards the competency can also be recorded. 
Competencies and criteria are not necessarily of the same order. A competency generally describes 
an act, behaviour, or performance. A criterion can describe a quality or element of performance. 
It is most likely that adult literacy and numeracy teachers will develop more and more sophisticat- 
ed descriptors of criteria for contextual and encompassing performance-based assessment activi- 
ties. The National Framework for Adult English Language, Literacy and Numeracy Competence 
(ACTRAC, 1 993) already provides a rich framework from which aspects and levels of performance 
can be drawn and the National Reporting System (NRS) (Coates et al., 1995) provides an equally 
rich system to which outcomes can be mapped. It is envisaged that in the future assessment exem- 
plars will provide further elaborations of appropriate criteria, descriptors, and standards. 
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Implications of changes for teachers and program planners 

The discussions thus far indicate that assessment in the Australian adult literacy and numer- 
acy context is an area that needs serious attention. Accountability to funders, institutions, 
other teachers and students is an expectation which at first glance may appear a bureaucratic 
imposition, but which in practice will ensure quality provision. From an instructional per- 
spective, accountability for student learning is best demonstrated through the use of appro- 
priate, sensitive and diverse assessment. 

Good assessment must be integrated with curricula that reflect the comprehensive definitions 
of literacy and numeracy accepted today. The curricula deal with all aspects of learning, not 
just objectives related to the acquisition of reading, writing, or computation skills. Curricular 
goals should include motivational, attitudinal, cognitive and metacognitive components. 
They should take into account the cultural backgrounds, prior knowledge and experiences of 
the students, but with the proviso that the goals and assessment should match the needs of the 
student learning group. Teachers should not presume on behalf of students. For example, 
students whose goal is to complete a Public Service Examination will need to develop ‘stan- 
dard’ Australian English, even as a second language, in order to be able to complete a formal 
and timed test. Assessment in this context will include diagnostic and formative assessment 
to guide effective instruction in the acquisition of ‘standard’ English, but should also include 
test-taking practice. 

Good assessment has to be congruent with the resources and materials that are found in class- 
rooms or training sites. A range of genres, materials with different difficulty levels, and rel- 
evant forms and documentation should be incorporated into the assessment tasks. Materials 
should be selected so that students can make use of their prior knowledge, as well as their 
awareness of the conventions of texts. Support materials such as dictionaries, thesauruses, 
calculators, and the like should also be available in assessment situations. 

A documented and planned approach towards assessment is essential for good assessment. 
Not only will this ensure that assessment is systematic, appropriate and matched to curricu- 
lum goals and resources, but also ensure that teachers have opportunities to reflect on their 
instructional and assessment practices. Such reflection allows teachers to evaluate what they 
are doing and to review learners’ goals so that modifications can be made. Both planning and 
reflection assist with the development of meaningful descriptions and profiles of students’ 
learning for reporting procedures. Documentation and planning of assessment provide mech- 
anisms by which teachers can discuss assessment issues with each other. 

We do not underestimate the demands and time consuming nature of these types of assess- 
ment practices. However, assessment plays a significant role in the work of the adult literacy 
and numeracy teacher. It is estimated that in primary and secondary schools, teachers spend 
twenty percent of their time in assessment activities. Most of the effort for teachers in devel- 
oping improved and effective assessment in the adult literacy and numeracy sector will be in 
the initial phases during which innovative and comprehensive assessment strategies are 
planned. Thereafter systems and procedures can be put into place which ensure that assess- 
ment becomes part of the routine programming cycle. 

Teachers can lighten the load by sharing planning, materials and assessment tasks with each 
other. They should also seek forums where they can discuss matters relating to assessment 
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and gain information about the best and most recent initiatives in adult literacy and numeracy 
assessment practice. The action research projects presented here describe investigations of assess- 
ment issues that concern teachers, the outcomes of the projects, and their authors’ deliberations and 
conclusions. This publication is one attempt to try to share, and lighten the load of teachers by dis- 
cussing innovations and considerations in assessment practice. 

Issues in assessment for the future 

During this discussion we have touched on a number of issues that are already confronting teach- 
ers in adult literacy and numeracy sites. However, several other issues are beginning to emerge or 
will need to be addressed in the future. 

Firstly, the implementation of assessment practices that we have described, such as continuous 
assessment and folio approaches to assessment, require the development by teachers of compre- 
hensive systems of documentation. The National Reporting System (Coates et al., 1995) to be used 
in federally-funded programs will also require documentation of outcomes. Teachers will have to 
explore how efficient documentation can be balanced with their workloads. 

As we noted previously, the direction of assessment towards performance-based and folio assess- 
ment introduces requirements to moderate work for comparability. The role of reporting mecha- 
nisms such as the National Reporting System is to allow common reporting frameworks for differ- 
ent types of provision and to accommodate different types of learners. Such systems are not meant 
to specify common assessment tasks. Therefore, in order to be able to relate assessment activities 
and outcomes, teachers need to be able to judge the equivalence both of assessment tasks and of 
ratings they give to performance on those tasks. Moderation is regarded as the most suitable mech- 
anism by which this can occur. Moderation is a procedure whereby teachers, and students if appro- 
priate, discuss together examples of assessment tasks and student performance to consider their 
equivalence. Moderation can occur within one provision site and across sites. For example, two 
teachers or more could get together to examine how they came to their judgements of a student’s 
performance based on folio contents. Although moderation is being introduced for many general 
certificates of education few teachers have had experience in this area. These skills will need to be 
developed and refined in the future. 

Thirdly, the introduction of new and more comprehensive systems of assessment will create a 
diverse set of questions regarding the ethics of assessment at the assessment task, student and insti- 
tution levels. A teacher will always have to make decisions about the appropriateness of assess- 
ment tasks for adult literacy and numeracy students. This requires thinking about equity, fairness 
and the consequences of the tasks. Students need to be aware that they are being assessed and of 
the purposes of assessment. How the information about their achievement will be used should also 
be provided to students, as well as the outcomes of students’ assessments. These matters require 
that literacy and numeracy providers examine how such information will be made available to stu- 
dents in a moral and socially just manner. 

A growing press towards cross-institutional comparisons, particularly for funding purposes, will 
add new strains to the ethical responsibilities of teachers and the administration of institutions. An 
emphasis by all teachers on quality provision, and quality assessment, substantial documentation 
as a basis for clear reporting, and moderation should help to avoid inequitable comparisons. We 
will always need to remind funding bodies that the adult literacy and numeracy students in great- 
est need of instruction may make very small gains at various times. Yet it is with these students 
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that the outcomes will eventually be most beneficial for our society and ourselves. It will not 
be an equitable or sensible outcome if funding is directed only at those who are high achiev- 
ers. 

We share the concerns of the adult literacy and numeracy communities that the increased 
emphasis on workplace, vocational and training programs, and associated accountability and 
reporting, will further displace community provision in adult literacy and numeracy. A real 
danger exists that the essential personalised, individualised characteristics of community pro- 
vision will disappear with such a push. Guidelines need to be made explicit for community 
providers about the extent to which they will need to conform to accountability and assess- 
ment requirements. 

Finally, in all areas of education, the need for both preservice and inservice professional 
development in assessment theory and practices in literacy and numeracy is great. 
Authorities should not expect such changes as have occurred to be implemented without 
intensive and adequate professional development that is periodically updated and reviewed. 
Through our own assessment workshops in 1 995, we have become aware of the demand. The 
commitment by teachers to undertake such professional development even within their own 
time, and requiring considerable travel, is evident. State departments of education and voca- 
tional training, universities and training institutions must be responsive to these needs. 

Conclusion 

In this chapter, we have only touched briefly on a number of significant factors affecting 
assessment theory and conceptions of literacy and numeracy, as well as the impact of these 
for adult literacy and numeracy assessment and reporting practice. Our goal has been to show 
that, while in the midst of dramatic and awesome changes, it is a time of opportunity to 
explore exiting and innovative assessment practices that enhance both the effectiveness of 
adult literacy and numeracy instruction and the demonstration of what students can do. 

The following chapters reflect the opportunities taken by adult literacy and numeracy practi- 
tioners to explore some of these issues through action research. 
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1 Diagnostic assessment used to indicate the student’s strengths and weaknesses and particu- 
larly to set directions for instruction, may incorporate both psychologically and educationally- 
based assessment procedures. Teachers must always be wary, however, of limitations in per- 
formance on a task affected by decontextualisation. 

2 Viewing refers to the ability to construct meaning from visual texts, to interpret visual symbols 
and conventions, and to understand and critically evaluate the relationships between texts, 
viewers, contexts, and creators of texts. 
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ASSESSMENT AS AN ONGOING FEATURE 
OF THE LEARNING ENVIRONMENT 
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ABSTRACT: This study looks at the change in assessment practices within the adult liter- 
acy field in response to wider training reform agendas, and the tensions created between 
required practice and attitudes about assessment inherent to the field, in particular in regard 
to more formal assessment practices. A postal questionnaire about assessment practices and 
attitudes received 37 responses from adult literacy practitioners throughout Queensland, and 
demonstrated that the field has developed a wide range of assessment practices, both formal 
and informal, though attitudes regarding formal practices are still largely negative or guard- 
ed. Professional development specifically in assessment strategies is shown to be the most 
influential factor in facilitating die transition between the traditional and the new accredited 
cultures of provision. 
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1. INTRODUCTION 

1.1 A CHANGING SCENARIO 

The face of adult literacy provision has changed substantially over the last fifteen years, as 
it has been gradually drawn into wider training and economic agendas. Traditionally, adult 
literacy occupied a marginalised niche characterised by voluntarism, and driven by social 
justice concerns (Dymock 1994). With growing recognition of the role literacy and numer- 
acy competencies play in industry training, or as a barrier to employment, adult literacy has 
won a place on centre stage of the mainstream national training agenda. The spotlight has 
brought the benefits of additional funding, and research effort, but there has also been a 
compromise of the field’s independence with the imposition of mainstream agendas, such 
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as accreditation of curricula, accountability to funding sources and the introduction of competen- 
cy-based training. 

The area in which these wider agendas have had the most obvious impact is on assessment prac- 
tices and reporting requirements. Falk (1995) is critical of the field’s inability to separate 
‘assessment’ from ‘reporting’ functions, implying that they should be viewed as separate 
domains. Whether or not this lack of distinction is problematic is open to debate. The practice 
of reporting to whomever necessitates assessment, that is, that a judgement be made on what 
learning has been taken place, with perhaps some predictive sense of what the student will subse- 
quently be capable of. For the purpose of this study, assessment and reporting will be treated as 
interdependent functions (see also Cope et al. 1995: 4-5). 

Formerly, adult literacy provision was characteristically in programs that were student-centred 
and needs-based with an underlying social welfare philosophy of promoting students’ self-confi- 
dence. These individualised programs were assessed or informed by informal negotiated evalua- 
tion of objectives, progress and content (Falk 1995: 17). The purpose of assessment was to 
inform the learning partnership of teacher and student, with generally no accountability require- 
ments to outside parties. 

These practices contrast sharply with the expectations of the current Department of Employment, 
Education and Training (DEET) funded literacy programs for unemployed clients, whereby 
providers must report back to a third party with vested interests - the funding agency who com- 
missioned the course. For this purpose, outcomes of individual students are mapped against a 
supplied profile of competence. Cope et al. (1995: 42) characterise these judgements as ‘high- 
stake’ assessments, whereby ‘unreliable assessment ... can have serious consequences for the life 
chances of the individual concerned’. Competency-based curricula explicitly specify learning 
outcomes and performance criteria at a range of levels. Though flexible to a degree in imple- 
mentation, these curriculum-driven, externally accountable programs are quite different in spirit 
and practice from the former responsive adult literacy provision. 

Adult literacy provision has many faces, ranging from local community-based provision typically 
by volunteer tutor scheme, learning support and integrated provision in SkillShare centres, to 
increasingly professionalised provision based in TAFE colleges and private training providers. 
Not all sectors are currently subject to rigorous assessment expectations. However, with the 
development of the National Reporting System to commence in 1997 amongst nominated sectors, 
the plan is to gradually include all sectors so the field can share a common language and report- 
ing tool to enhance articulation and portability. Thus all providers will eventually face scrutiny 
of their assessment/reporting processes. 

To date, these changes have largely been imposed from above, as opposed to developing from 
within the field. Other external agendas have effectively set the tone and direction for develop- 
ments. This resulted in what Sanguinetti (1994:3) terms ‘confronting discourses’. 

Discourses of competitiveness and human capital theory challenge ‘social justice’ discours- 
es; discourses of efficiency and competency-based training confront discourses of critical 
literacy, progressivism and holistic, learner-centred pedagogies. 

The external agendas are not necessarily informed by an understanding of the nature of language 
learning or literacy development and can create real conflicts for the provider. 
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Teachers are confronting the conflict between their pedagogical practices and commit- 
ments and the requirements of competency-based assessment which are constructed 
within a very different set of discourses that those which have constructed their peda- 
gogical understandings and practices in the past. (Sanguinetti 1994:41) 

This raises the question of how these ‘conscripts’ are coping in their changing environment. 

1.2 THE EVOLUTION OF ASSESSMENT IN ADULT LITERACY 

Past attempts to institute shared reporting frameworks in adult literacy, and their reception 
by the field, demonstrate a gradual if grudging acceptance over time of such requirements 
(see Cope et al. 1995: 23-32 for a chronological description of the various stages). Though 
not designed as assessment instruments, but rather tools for reporting outcomes, profile 
frameworks such as the ‘Global Profiles’ and the Adult Literacy and Numeracy 
Competency (ALAN) Scales were highly controversial when proposed. The heated 
response they evoked was possibly more to do with the general issue of whether or not the 
field was willing to assess their students for an external audience, than with the relative 
merits of each tool. Their rejection led to the creation of the Interim Literacy Curriculum 
Matrix (ILCM) in 1991 to map curricula and aid student placement in labour market pro- 
grams. It is significant that the National Reporting System will replace this ‘interim’ mea- 
sure in 1997, only after the profession has had time to actively contribute to the reporting 
agenda, and respond with a framework that is informed by, and compatible with, pedagogi- 
cal principles of the field. 

While the field has been gradually convinced that it is possible to distil a workable generic 
profile schema to suit various reporting audiences, the parallel development of competency- 
based training (CBT) policy has also made its impact. Competency-based training in gener- 
al is characterised by its criterion-referenced assessment methodology. 

Assessment in competency-based training system has formalised the relationship 
between curriculum and assessment. Assessment is fundamentally linked to the com- 
petencies in the curriculum. (Cope et al. 1995: 48) 

In practice, the move to CBT curricula has meant a forced departure from holistic, negotiat- 
ed assessment practices that captured non-linguistie outcomes, such as improved student 
confidence and motivation, and the move to a detailed analysis of student competencies 
against specified learning outcomes. 

There is considerable tension between the ‘conflicting discourses’ of competency-based 
training and literacy development. There is debate as to whether it is necessary to fragment 
or itemise literacy competencies to the degree to which current curricula such as the 
Victorian Certificate in General Education for Adults (CGEA) have, or whether it is possi- 
ble to devise and assess against more generic competencies that reflect the pervasiveness of 
literacy practices (see Sanguinetti, 1995:41). Competency-based training implicitly oper- 
ates on a serial or step model of learning whereby the learner accrues a collection of com- 
plementary competencies. In contrast, language learning and literacy development curricu- 
la favour a spiralling model where base line understandings are constantly revisited and 
developed to new levels of complexity and sophistication. Regardless of whether CBT 
practices are compatible with language learning principles, the move to competency-based 
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curricula has focussed attention on assessment as a feature of the learning environment and 
drawn literacy provision into the mainstream training agenda. 

1.3 CURRENT ASSESSMENT PRACTICES 

For the purposes of this study, assessment was defined as any process or procedure whereby 
judgments are formed of a student’s degree of competence, strengths or weaknesses, or of his/her 
progress towards the objectives of a learning program. 

This definition thus includes both ‘formal’ and ‘informal’ assessment techniques. These terms 
are widely used but their boundaries are often blurred in application. Falk (1995) used the terms 
but did not attempt to define them, though he concludes that there is a disappointing equating of 
‘assessment’ with ‘formal assessment’ amongst his respondents, who generally dismiss or fail to 
recognise informal assessment practices. 

Adult literacy practitioners have always been involved in assessment, in particular, initial diag- 
nostic assessment, and ongoing formative assessment for program negotiation and student feed- 
back. These assessments were typically informal, in that they avoided test-like situations, or exit- 
point assessment that were enshrined in scores or ranking. This avoidance was usually fuelled by 
a concern to protect the budding self-confidence of the student from any sense of failure or inad- 
equacy. 

To capture the full range of assessment practices, including the informal, in this study we 
attempted to distinguish formal and informal assessment explicitly as follows. 

Formal assessment takes place when: 

• assessment is planned and undertaken consciously by the teacher; 

• the outcomes are systematically recorded, to inform any reporting requirements; 

• the same assessment is applied to all students in the program, under similar conditions; 

• students are aware that they are being assessed. 

Informal assessment takes place when: 

• ongoing cumulative judgements and feedback on student progress are derived from inciden- 
tal observation and participation in the learning activities of the program; 

• the outcomes may not be recorded, but add to the teacher’s incremental mental picture of 
the student’s performance; 

• students may not be aware that judgements are being made; 

• assessment judgements are made in contextualised interactions, which may differ between 
students. 

By these definitions, formal assessment would include test or exam-like procedures, but also sys- 
tematic collection of samples of work, targeted and recorded observations of student perfor- 
mance, and recorded self-evaluations by students. ‘Formal’ assessment is distinct from ‘stan- 
dardised’ testing, which is statistically validated testing against established norms, independent of 
the learning context. 

Falk (1995), in his evaluation of assessment practices in adult literacy provision, was critical of 
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the limited extent and quality of the assessment practices observed. He reported a lack of 
overall cogent assessment regimes, an imbalance between the considerable effort devoted to 
initial diagnostic testing, and the paucity of formal summative feedback to students. He 
also considered that the formal assessment that was happening was over-reliant on outdated, 
decontextualised skill testing. He attributes these deficiencies to inadequate resourcing, the 
increasing casualisation of literacy employment, and recruitment from school -trained teach- 
ers. He reports positive developments in addressing assessment in competency-based pro- 
grams, though he feels such CBT assessment can suffer from simplistic interpretation. 
Though he criticised the providers for devaluing informal methods such as the ongoing 
looking over the shoulder (1995: 220), the mini-study of teacher judgements was disap- 
pointingly curtailed and inconclusive. In addition, his survey instruments tapped conscious 
and planned assessment regimes, and did not really encourage examination of the informal, 
less conscious data gathering that informs teacher judgements. 

Cope et al. (1995) highlight the role moderation practices can and should play in assess- 
ment strategies, for intra- and inter-program purposes. Moderation networks were seen to 
act as safeguards to ensure the reliability of assessment processes, and perhaps more impor- 
tantly, as venues for constructive criticism and collaboration in the face of imposed change 
from above. Sanguinetti (1994: 36) reports that some teachers found the exposure threaten- 
ing, but the reports stress that moderation processes are an essential part of any assessment 
strategy. 

1.4 ATTITUDES TO ASSESSMENT 

While the assessment duties required of adult literacy practitioners have changed, have their 
underlying attitudes changed in accordance? Are they converted and willing participants in 
the new assessment regimes, or are they reluctant participants who are yet to be convinced 
of the validity and benefits of more formalised assessment and reporting formats? How 
does any conflict between attitudes and designated duties affect their assessment practices? 

The literature suggests that there is considerable friction between attitudes and required 
assessment practices. Falk (1995: xvi) describes a pervading negative attitude to assess- 
ment, based on outdated perceptions of what constitutes assessment. 

Because of the negative feelings towards assessment, derived ... from unfortunate per- 
sonal recollections of formal examinations and testing on behalf of the teachers and 
co-ordinators, staff appeared to explicitly avoid dealing with assessment as a relevant 
educational process. 

Sanguinetti (1 994) reports that there is considerable frustration amongst adult literacy 
teachers accountable to the Victorian CGEA, who have found their programs hijacked or 
driven by the CBT assessment agenda, and that the system disallows the intuitive assess- 
ment practices that in the literacy practitioners’ view constitute their professionalism (1994: 
21-24). In addition, there is dissatisfaction with the ascribed reporting tools, and their fail- 
ure to depict the diversity or degree of learning achieved (Cope et al. 1995: 39). 

This gap between attitudes and practices is possibly wider for the adult literacy teachers 
than for teachers of English as a Second Language (ESL), given that the former’s common 
rationale for resistance to formal assessment is the concern for feelings of the students 
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undergoing assessment (Falk 1995: 128). This in turn stems from the social welfare philosophy 
underpinning the development of the field. The ESL student body will also include people with 
little or no formal education, but in addition will draw from people with varying amounts of edu- 
cation in their first language, and possibly different cultural expectations about the nature of 
learning and teaching. Cope et al. (1995: 22) cite research that shows that ESL students in gener- 
al want to participate in formal assessment. Unfortunately, there was no mention of parallel 
research for adult literacy students. Hence the widespread assumption that they do NOT want 
formal assessment, and that formal assessment would in fact be detrimental to their learning 
process, remains unchallenged. 

1.5 THE ISSUES BEHIND THIS STUDY 

Dymock (1994: 64) sums up the dilemma that many practitioners now face on a daily basis: 

Being in the mainstream also means swimming with the tide - we need to be willing to go 
in the direction the tide is carrying us or be strong enough to swim against the current when 
we believe the occasion warrants it. 

It was this complex interplay of factors and developments - external agendas creating conflict 
between the attitudes and practices of practitioners as they dance to these new tunes - that raised 
the questions behind this research. Just what assessment practices are being employed through- 
out the field? How do practitioners feel about assessment as an ongoing part of their programs? 
How are changing policy and requirements affecting teachers at the workface? Where are the 
tensions between attitudes and practice? What are the characteristics of practitioners who are 
comfortable (or uncomfortable) with more formal assessment requirements? What training have 
practitioners in the field had to facilitate the transition between paradigms? How well equipped 
do they feel? 

2. METHODOLOGY 

A postal questionnaire (Appendix 1) was developed with three general sections - characteristics 
of the practitioner, current assessment practices, and attitudes towards a number of issues con- 
cerning assessment. Some questions included pre-coded responses for quantitative analysis. 

Other open-ended questions invited comments for qualitative data. The questionnaire was piloted 
with part-time adult literacy staff at the Toowoomba College of TAFE. 

In light of Falk’s (1995) concern that only formal assessment strategies were identified as assess- 
ment, definitions of ‘assessment’, ‘formal assessment’ and ‘informal assessment’ were included 
at the beginning of the questionnaire. To further ensure that the full range of assessment strate- 
gies were captured in the data, the relevant section included a substantial list of possible assess- 
ment methods. Respondents were asked to indicate which of these they practised, and to 
describe any additional assessment methods they employed. 

The respondents were identified through the six Queensland Adult English Language, Literacy 
and Numeracy Regional Networks throughout Queensland. These networks were set up as part 
of Queensland’s response to the National Collaborative Adult English Language and Literacy 
Strategy to coordinate and enhance the varied efforts in adult language and literacy. The net- 
works include representatives from stakeholder groups, including providers in TAFE, SkillShare, 
private provided and community programs. The Regional Executive Officers that coordinate 
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these networks were asked to nominate practitioners from all of these sectors, including 
full-time teachers, part-time teachers and volunteer tutors. 

A total of 55 practitioners spread throughout Queensland were invited to respond to the 
questionnaire within a two-week time frame. A reminder letter was set out one week after 
the questionnaires. Thirty-seven responses were received, an encouraging response rate of 
67%, which in itself suggests that the research is timely and touched on some sensitive 
points that are of current concern. One response was a collective response from a group of 
teachers, and could not be included in the quantitative analysis. 

Quantitative date were compiled and analysed using Statistical Package for the Social 
Sciences (SPSS) for Windows. Frequencies were calculated, and Pearson Chi-Square tests 
were carried out on a range or cross-tabulations of response items. For the attitudinal 
responses, the five-point range of responses was recoded to a three-point (positive, unsure, 
negative) scale to enhance the Chi-Square tests. Given the small sample size, weak associ- 
ations between variables that are not statistically significant but may indicate a trend have 
been included in the discussion where of interest. Some qualitative responses, for example 
descriptions of assessment training or inservice, were coded into categories for quantitative 
analysis. Other qualitative data were collated and examined for themes, and issues. 

3. FINDINGS 

3.1 RESPONDENT AND PROGRAM CHARACTERISTICS 

The vast majority (92%) of the respondents were 36 years or older, the largest cohort being 
between 36 and 45 years of age. Twenty-nine (91%) of the respondents were female, 
reflecting the common bias in the field. 

Nineteen (53%) have been working in the adult literacy field for more than 5 years. 

Another 15 (42%) have between one and five years' experience. Thirty-two (89%) of the 
respondents had experience in other educational contexts: 21 in primary education; 13 in 
secondary; 4 in remedial/disability programs; 2 in ESL programs; and 15 in adult education, 
including vocational training, bridging education and tertiary settings. Many had experi- 
ence in more than one sector apart from adult literacy. 

Thirty-three (92%) of the respondents had teaching qualifications (certificate, degree or 
diploma). Of these 14 (39% of all respondents) had pursued or were currently studying 
specialist studies in literacy or language education, including post-graduate qualifications, 
or the accredited inservice courses, ‘Adult Literacy Teaching' and ‘Adult Numeracy 
Teaching'. Three (8%) of the respondents had volunteer tutor training, or workplace train- 
ing qualifications. 

Thus, in general, the literacy workforce sampled is mature, female, experienced and profes- 
sionally qualified, with a significant number pursuing or in possession of specialist qualifi- 
cations in the area. They also display a significant breadth of experience in a range of edu- 
cational settings. 

Twenty-one (58%) of the 36 respondents had had a moderate level of training specifically 
in adult literacy assessment. A moderate level was interpreted as including inservice 
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programs, such as Adult Literacy Teaching (ALT), or Adult Numeracy Teaching (ANT), with 
elements devoted to assessment methodology. Nine (25%) had had a minimal level Oust as part 
of their volunteer tutor training) or no training, and 6 (17%) had had a substantial level (a tertiary 
unit or post-graduate study devoted to adult literacy assessment). The bulk at the moderate level 
could reflect the recent concerted effort to inservice adult literacy providers throughout the TAFE 
sector in particular. To a lesser extent, other providers have also accessed the ALT and ANT 
courses when available. 

When asked to detail any other training that has contributed to their general understanding of 
assessment, 14 (39%) could not think of any such training, while 16 (44%) felt that their initial 
teacher training, and professional development programs in previous educational contexts con- 
tributed to their understanding. Six (17%) nominated previous training in remedial/disability 
education or in teaching English as a second language. 

The sample included 4 part-time teachers, 19 full-time teachers, 1 volunteer tutor, 7 tutor coordi- 
nators, and 5 literacy workers who fell outside these categories, with roles such as SkillShare 
training officer, prison education officer, program coordinator or manager. This does not reflect 
the widespread trend of increased use of casual part-time teachers to staff literacy programs. 
Given that the nominations came from the networks’ membership, full-time staff are probably 
better represented in this consultative activity than part-time or volunteer staff. There is a chance 
that given the sample’s bias towards full-time staff, factors such as qualifications, inservice train- 
ing and participation in assessment moderation may be overstated. However, given that the part- 
time staff, their attitudes and assessment practices may well set the tone and direction for the pro- 
grams under their management, and thus their responses could be adequately representative of 
wider practice. 

As perhaps to be expected, the more highly qualified practitioners reported more training specifi- 
cally in adult literacy assessment (p < .01). This also suggests that assessment is not a focus in 
lower levels of training, such as volunteer tutor training. Full-time workers displayed significant- 
ly higher levels (moderate to substantial) of training specifically in adult literacy assessment (p < 
.05). Those with substantial training in adult literacy assessment could also identify moderate to 
substantial levels of other relevant assessment training in their background that informs their cur- 
rent practice (p < .05). 

Part-time teachers and tutor coordinators had minimal to moderate levels of specific adult literacy 
assessment training. This highlights one possible risk of the increasing casualisation of literacy 
employment, that is, that part-time employment and dubious career prospects will not be enough 
incentive to pursue further study and qualifications in the field. Four of the 7 tutor coordinators 
only had minimal levels of training. This proportion could be of concern when this mode of pro- 
vision is required to participate in more formal reporting structures, for example the National 
Reporting System (NRS). As the coordinators are the major professional support to the volunteer 
workforce, this sector could require considerable development prior to the NRS implementation. 

Twenty-three (64%) of the respondents worked in TAFE programs, 7 (19%) in SkillShare set- 
tings, three (8%) in community-based provision and one (3%) in a workplace program (though 
other respondents also worked sometimes in workplace programs). This spread over the various 
sectors of provision is uneven and makes it difficult to draw decisive conclusions about differen- 
tial practices across sectors, but comparisons of the data obtained may be indicative of patterns. 
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In the sample, 17 (47%) described their communities as regional urban centres, 15 (42%) 
described their community as rural, and only two responses came from metropolitan commu- 
nities. Another two respondents felt their communities fell outside these categories - one 
working in a prison program, the other servicing an industrial/rural mix. The sample there- 
fore is skewed towards providers outside the metropolitan areas. This could raise the ques- 
tion of whether degree of access to professional development opportunities and decision- 
making processes could affect attitudes and practices. On the other hand, the sample has 
managed to extend beyond the frequently sampled metropolitan setting, to reflect the diverse 
faces of literacy provision. 

Of the 36 respondents, 27 (75%) were operating within a curriculum which specified target 
learning outcomes. Of these, 22 (61% of the total sample) were required at the end of the 
program to supply a report on each student against these learning outcomes. This reflects a 
major shift from negotiated student-centred programs to accredited curricula, and account- 
ability to funding agencies. The vast majority of TAFE programs reported the use of explicit 
curriculum (20 of the 23). SkillShare providers seem to be in a state of transition, three of 
the seven sampled using explicit literacy curriculum. Programs with explicit curricula dis- 
played a higher level of specific assessment training (p < .05), suggesting that the introduc- 
tion of accredited curriculum has driven the professional development program, and deliv- 
ered training where it is needed. However, there were still four respondents working under 
explicit curriculum with only minimal levels of assessment training. 

3.2 ASSESSMENT PRACTICES 

The following table outlines the number of respondents who indicated that they used the 
described assessment practices. 

Table 1: Number of respondents employing specific assessment practices (N=36) 



ASSESSMENT PRACTICE 



FREQUENCY (%) 
32 (89%) 



A. Initial diagnostic assessment for placement and/or program planning. 

B. Ongoing informal observation and consideration of 
my students’ progress, strengths and weaknesses. 

C. Systematic recording of my observations about my students’ progress 
on an ongoing basis. 

D. Regular tests of word recognition and phonic skills. 



20 (56%) 
15 (42%) 



34 (94%) 



E. Maintenance of a portfolio of samples of students’ work to demonstrate 
progress. 

F. Encouragement of self-evaluation by the students throughout the 
program. 

F. Facilitation of self-evaluation by students of their overall progress on 
completion of a course. 

H. Regular tests on agreed list of number facts or words 
(for spelling or vocabulary). 

I. Formally administered assessment tasks such as tests that reflect 
the program objectives, as checkpoints through the course 
(formative assessment). 

J. Formal assessment at the end of the course to assess overall student 
progress against stated outcomes, to facilitate future placement or 



27 (75%) 



24 (67%) 



22 ( 61 %) 



30 (83%) 



13 (36%) 
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to fulfil any reporting requirements (summative assessment). 13 (36%) 

K. Participation in moderation of students’ performance on assessment tasks 

with other literacy or numeracy teachers. 20 (56%) 

L. Matching of students’ performance on assessment tasks to some supplied 
descriptions or profiles of competency levels (e.g. ALAN Scales, or the 

Interim Literacy Course Matrix/ILCM). 20 (50%) 

M. Contribution to other staffs assessment of my students’ progress. 18 (50%) 

N. Other. 6 (17%) 



Examples of ‘Other’ practices included: 

• informal, formative moderation; 

• remote assessment by School of Distance Education; 

• liaison with volunteer tutors re student progress; 

• self assessment prior to the course; 

• regular discussions with course teachers to evaluate students’ progress. 

Most of these are compatible with Items K or M and could have been subsumed under them. 

The most obvious conclusion to be drawn from these data is that adult literacy practitioners are in 
fact busy assessing, and employ a wide range of strategies, averaging a combination of more than 
eight methods each for assessing their students’ progress. 

The less popular methods, Items in D and H, are precisely the ‘older style, decontextualised skills 
tests’ that Falk (1995) criticised the field for over-utilising. This study then suggests a shift away 
from these practices. Without entering the debate of their relevance or pedagogical worth, it is sig- 
nificant that any respondent who utilised such methods also nominated a range of other methods of 
assessment. 

An overwhelming majority (94%) indicated that they used ongoing informal observation as part of 
their assessment strategy. Falk’s suggestion (1995: xvi) that there was ‘a lack of explicit aware- 
ness of the role and nature of informal during course assessment ...’ is not supported by these data. 
Rather, informal strategies were found to be ranked as the most commonly employed strategy, 
more common than the other highly ranked item - initial diagnostic assessment. The only two 
respondents who did not report this practice were tutor coordinators who presumably were one 
step removed from the learning context. This shows that traditional assessment practices have not 
been replaced or superseded, but rather supplemented with additional strategies. 

Self-evaluation as formative and summative assessment strategies (Items F and G) were also popu- 
lar strategies employed by 83% and 61% respectively. The decreased number for summative self- 
assessment could possibly be explained by the funding bodies’ rejection of these as valid assess- 
ment tools for their purposes. Formative self-evaluation was particularly popular with full-time 
teachers (18 of the 19). Eighteen of the 27 programs that operate under explicit curriculum report- 
ed the use of summative self-evaluation strategies. These figures contrast with Falk’s findings that 
reported a ‘prevalence of ‘top-down’ practices’ (1995: xvi). 

These disparities between the two surveys could be the result of different samples, different survey 
instruments, or a change in practice over the intervening years. In support of elements of Falk’s 
findings, the vast majority of practitioners surveyed have indicated that they undertake initial 
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diagnostic testing (Item A). Similarly, in accordance with Falk’s findings, formal summa- 
tive assessment (Item J) is carried out by just over a third (36%) of the respondents. 
However, formal formative assessment (Item I) is carried out by 67% of the respondents. 
Formal formative assessment was overwhelmingly used in programs with an explicit cur- 
riculum (p < .01) and reporting requirements (p < .01). This suggests that change to more 
formal assessment practices is being driven by top-down agendas such as the introduction 
of accredited curriculum. In addition, formal formative assessment was only reported by 
respondents with teacher training or specialist qualifications. 

Formal summative assessment (item J) was not reported in any programs which don’t have 
an explicit curriculum, and in fewer than 50% of the programs that do. Even amongst those 
programs with reporting requirements, the majority (13 of 22) did not use formal summa- 
tive assessment. Fifteen of the 23 TAFE programs samples did not use formal summative 
assessment. 

There are two possible explanations for the drop off in formal assessment at the summative 
stage. Firstly, Cope et al. (1995: 18) describe what they term ‘ongoing summative’ assess- 
ment practices in competency-based courses, whereby various learning outcomes are 
assessed along the way, to create a composite summative assessment by the end of the 
course. In this light, the high response in Item I could indicate this sort of practice. 
Secondly, the current Queensland TAFE system has, to this point, retained the option for 
adult literacy students to choose not to participate in final evaluations. Whether this choice 
is mediated by teacher attitudes and advice, or whether it is genuine and independent reluc- 
tance on the part of the learners, the drop-off in formal summative assessments may be par- 
tially accounted for in this way. 

To return to the distinction of ‘formal’ and ‘informaT methods as defined previously, Items 
C (recorded observations), D (word/phonic tests), E (portfolios), M (number/spelling tests), 

I (formal formative), J (formal summative), and L (profile mapping) assessments could all 
constitute formal assessment in that they are conscious, planned, systematic and document- 
ed. Other practices such as self-evaluations (Items F and G), moderation (Item K) and 
diagnostic assessments (Item A) could also be carried out in a formal or informal way. The 
incidence of such formalised practices suggests that the field has largely come to terms with 
more formal, visible assessment processes: 

Twenty respondents (56%) indicated that they participate in moderation processes (Item K). 
This is a slender majority, and it is disappointing that not more practitioners can benefit 
from this valuable networking and collegial process. Sixty-seven of the sampled programs 
with explicit curriculum (18 of 27) reported participation in moderation processes. To cater 
for more low key informal moderation strategies, Item M, ‘contributing to other staff/s 
assessment of my student’s progress’, was included. Of the 18 respondents (50%) who 
reported this strategy, 16 were working within explicit curriculum. Interestingly, the one 
volunteer tutor and 5 of the 6 tutor coordinators did not report this strategy. 

Twenty (56%) respondents employ profiling instruments (Item L). Nineteen of these 
respondents are working with explicit curricula. Usage of profiles was only reported by 
practitioners with teacher training or higher qualifications, though not by all of these 
groups. A significant majority of programs with reporting requirements (16 of 22) used 
profile instrument (p < .05). There was limited use of profile instruments in other program 
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types. This indicates a growing, if not complete, acceptance of such instruments. The non-usage 
of profile instruments in other sectors of provision suggest that the blanket introduction of the 
NRS will require extra input. 

When examining which sub-groups favour which assessment practices, the data suggest that many 
strategies are frequently distributed surprisingly evenly over various respondent categories. For 
example, Item C, systematic recordings of teacher observations had no correlation with whether 
the program utilised an explicit curriculum, or required reporting on completion, or with the work 
status of the respondent. Similarly, the use of phonic or word recognition tests (Item D), the main- 
tenance of portfolios (Item E), and the use of spelling tests (Item H) showed no significant correla- 
tion with program type, training or qualifications of the respondent. These practices were not 
noticeably the domain of any sub-group. 

In an open-ended question, respondents were asked to describe assessment practices that they 
would like to use in the future. The following table summarises their responses. 



Table 2: Assessment practices respondents would like to develop by frequency (N=36) 



DESCRIPTION OF ASSESSMENT PRACTICE 


FREOUENCY 


A. 


Peer assessment 


1 


B. 


Systematic recording of observations 


3 


C. 


Student self-assessment 


3 


D. 


Development of assessment item bank to support a curriculum or program 


3 


E. 


Identification of student goals 


1 


F. 


Better initial diagnostic assessment practices 


5 


G. 


Use of profile instruments 


4 


H. 


Moderation 


3 


I. 


Formal summative 


1 


J. 


Involving and educating other parties 

(e.g. CES, industry program managers) in assessment 


2 



This broad spread shows a degree of curiosity in exploring, improving and extending assessment 
repertoires amongst the respondents. It should also be noted that there were three other responses 
in the nature of ‘no more please!’ 

When asked to nominate the additional resources, training etc. they would need to implement 
these desired practices, the responses included: 

• professional development opportunities (12 responses); 

• better and more resource materials, such as assessment kits, item banks, and checklists (7 
responses); 

• more time (2 responses); 

• support staff; 

• liaison with other providers; 

• qualified staff; 

• moderation meetings; 

• access to information about developments. 
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To confirm whether or not changes in policy have in fact filtered down to affect classroom 
practice, respondents were asked to describe what changes they have witnessed in assess- 
ment through their years of experience. 

Ten of the 37 responses received singled out the increased accountability requirements to 
funding bodies, often suggesting that these expectations are met at the expense of assess- 
ment for the purpose of student feedback: 

Accountable to more sources, e.g. CES, industry, management (provider), let alone 
clients! 

Assessment is a requirement of systemic/funding factors not for any leaming/adult 
education factors. 

A move towards quantification of performance - more for admin reasons than student 
benefits. Far more paper work required of teachers - however I see no benefit to the 
students - in fact the opposite in that I believe students tend to miss out on broader 
lit/num in favour of the set outcomes. 

Nine responses mentioned the increased usage of formal assessment practices, and the new 
primacy given to assessment: 

More expectation towards formal assessment as curriculum and QA (Quality 
Assurance) comes on line. 

Implementation of CNL03 - Certificate in Vocational Access, has formalised and 
structured assessment procedures. 

While it was once believed that formal assessment was the BIG no-no, current prac- 
tice places assessment also too much at the forefront. 

The move to a competency-based curriculum and its assessment demands were nominated 
by 5 respondents as a major change in practice: 

... the introduction of CBT has made a huge impact on assessment procedures. I think 
it is a good thing to clarify what exactly one is assessing, but I also think that CBT 
assessment can be quite reductionist. 

Switch to competency-based assessment in TAFE colleges, more accountability, more 
systematic reporting of progress, more structured courses. 

with positive developments: 

More rigour while maintaining the ‘supportive’ perspective. Competency/ outcomes 
provide goals to reach rather than barriers to overcome. 

Six other responses felt the move to more formal methods and structured curriculum had 
gone too far: 
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Has become too structured and does not meet individual learners’ goals. 

Pre -determined outcomes and assessment requirements offer little flexibility for special 
needs (of which the adult learners have a great abundance). 

Seven other respondents felt that the new focus on assessment was out of proportion: 

From nothing to the ridiculous! 

Too much! 

with unrealistic administrative demands placed on the teacher: 

Assessment has become a time-consuming function of the literacy teacher who now has to 
keep an inordinate number of records and who cannot for quality purposes fail to keep 
these records. 

There was another group of responses that mentioned the role professional development should 
have played in supporting the changes. 

Although I am aware of the move towards the formalisation of assessment procedures I feel 
that any implementation should be preceded by training and clear guidelines ... 

Curriculum needs in the area of requirement and expectation have been established by 
National training bodies, but the expertise of the average teacher to carry them out has not 
been developed by the necessary training. 

One respondent sums up the respondents’ view of changes they have witnessed in assessment 
practices: 

Increased formality/ training/ accountability/ administration/ ongoing changes and expecta- 
tions re staff expectations, needs, competence and study. 

33 ATTITUDES REGARDING ASSESSMENT IN ADULT LITERACY PROVISION 

To gauge practitioners’ attitudes to assessment issues, respondents were asked to indicate the 
degree to which they agreed or disagreed with a range of statements. The statements were not 
designed to reflect any one stance on assessment methodology, but rather to test the water on dif- 
ferent issues, such as informal versus formal practices, the adequacy of professional training, 
standardisation of assessment practices, and how students react to different assessment regimes. 
Table 3 (Overleaf) outlines the range of frequencies of responses to these statements. 
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Table 3: Attitudes towards assessment issues: Frequency of response and percentage of total 
respondents by trigger statement (N=36) 

Attitudinal Statement Strongly Disagree Unsure Agree Strongly 

Disagree Agree 



a. Formal assessment forms 

an important and conscious part 

of my teaching strategy. 9(25%) 10(28%) 6(17%) 8(22%) 3(8%) 

b. Informal assessment forms 
an important part of my 

teaching strategy. 2 (6%) 12 (33%) 22 (61%) 



c. I feel my training has equipped 
me well to undertake the program’s 

required assessment duties. 3(8%) 6 (17%) 1 1 (31%) 9 (25%) 6 (17%) 



d. Adult literacy students benefit 
from the clear indication of 

their levels and progress that formal 

assessment can give them. 1 1 (31%) 9 (25%) 10 (28%) 3 (8%) 2 (6%) 

e. Most adult literacy students 
are intimidated and discouraged 
by formal, structured assessment 

tasks, such as tests. 6 (17%) 4 (1 1%) 8 (22%) 18 (50%) 

f. Most adult literacy students are 
not daunted or discouraged by 
informal assessment procedures, 
e.g. interviews and classroom 

observations. 1 (3%) 4 (11%) 2 (6%) 12 (33%) 17 (47%) 



g. It is important to develop an 
assessment strategy at the 
same time as developing 

program objectives. 1(3%) 1(3%) 6(17%) 10(28%) 18(50%) 



h. Any adult literacy curriculum should 
supply standard assessment tasks 
with marking guides so all students 

are assessed in the same way. 12(33%) 7(19%) 6(17%) 4(11%) 5(14%) 



Note: Some rows do not total 100% because of the occasional missing response. 

Statement A displays a wide spread of opinion, with only 30% indicating that they value formal 
assessment strategies. Over 50% disagreed that formal assessment is an important part of their 
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pedagogy. There was no relationship between how the respondent felt and whether the respon- 
dent worked with an explicit curriculum, or the type of provider (TAFE/SkillShare/community). 
However, there was a significant association with the amount of specific training the respondent 
had had (p < .05) in adult literacy assessment. Those with minimal training in assessment were 
uniformly negative, those with moderate to substantial levels of training were spread across the 
range. There was also a weak trend suggesting that the more highly qualified practitioners were 
more comfortable and positive about the role of formal assessment. 

In stark contrast, 94% of respondents value informal assessment strategies (Statement B), sup- 
porting the traditional preference for non-intrusive assessment methodology in the field. The two 
respondents who did not respond positively indicated that they were unsure, as opposed to a neg- 
ative response. 

Of those who agreed with Statement A (‘formal assessment is important’), additional comments 
suggest that some feel forced into this position: 

Don’t like it this way, BUT we are forced into more formal assessment. 

Others believe that formal assessment experience equips students for further study settings. Of 
those who are unsure or negative about formal assessment, their opinion seems to be based on the 
belief that formal assessment demands a compromise with programming based on student needs: 

It is something which can take too much of the focus away from what a student wants and 
needs. 

There is another group who felt that circumstances, that is, the students involved and their level 
of competency, should be the deciding factor: 

I find formal assessment threatening and reminiscent of usually painful school experiences 
for the students. Some formal testing is gradually introduced corresponding to confidence 
growth. 

Comments in support of informal methods show how integral they are to adult literacy provision, 
informing constant feedback to the student and the program: 

Students are constantly made aware of any improvements. 

After each lesson, I plan future lessons using weaknesses noticed. 

This is an ongoing process informing what happens in sessions. 

Comments in favour of informal methods also imply a concern with the efficacy of other assess- 
ment modes: 

Observations, conversation will often give a much better indication of a student’s depth of 
understanding. 

Less pressure on students can give a more accurate reflection of skills. 
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Statement C invited an opinion on how well equipped practitioners feel to undertake their 
assessment duties. There is a considerable spread of opinion, the bulk being unsure either 
way, with a slight skew towards the positive end. This suggests a degree of uncertainty and 
lack of confidence amongst a significant number of adult literacy teachers when it comes to 
assessment, and should alert the field to the need for substantial professional development 
to support any blanket introduction of the National Reporting System. There was a highly 
significant association between the level of specific training in adult literacy assessment (p 
< .01) and the attitudes expressed here. The higher the level of such training, the more pos- 
itive the response. Higher levels of assessment training displayed a stronger association 
than did higher levels of qualification. Full-time practitioners were more positive than oth- 
ers (p < .05), but they, as a group, displayed higher levels of specific assessment training. 
Eleven of the 26 respondents operating under explicit curricula displayed uncertainty, while 
6 of the 9 working in negotiated programs felt positive about their level of training. 

Additional comments given highlight that experience is perhaps more important than train- 
ing opportunities: 

I believe my experience teaching in a number of different programs and institutions 
with vastly different program objectives has helped me to formulate assessment pro- 
cedures and practices appropriate to each situation. The recent training I completed 
reinforced much of my current thinking in this area. 

This perhaps reflects the sample’s breadth of other teaching experience. Other comments 
give the sense that teachers are constantly chasing the changing agenda, without any sense 
of arrival: 

Not really (feeling well equipped) in that during my time as an adult literacy teacher 
admin requirements re assessment procedures have/are continually changing. 

Given that the majority of respondents had worked in adult literacy for a number of years, 
this is probably a common feeling. 

In response to Statement D, 57% indicated that they were not convinced that formal assess- 
ment carried significant benefits for their students. Another 29% were unsure, with only 
14% confident that there were benefits in formal assessment for the students. This resis- 
tance to formal methods was widespread through the sample, with no significant associa- 
tions with other respondent characteristics. 

Some respondents clarified their ambivalence: 

Yes, if they are making progress. No, if their progress is in smaller or slower steps 
than the assessment items can register. 

This is true of some students. Many, however, find formal assessment stressful and 
often do not demonstrate a true level of their ability because they do not perform well 
under pressure. 

It depends entirely on their level of literacy. 
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Others were more cynical: 

No, the teachers benefit more! 

Statement E is the flip side of Statement D. There is a stronger indication here that the respon- 
dents believe that the risk of formal assessment is the potentially damaging impact on the stu- 
dents, 72% (50% strongly) agreeing that most students would be intimidated and discouraged. 

This feeling was spread throughout the sample, with no significant association with other factors. 
Additional comments, however, revealed another group of students who are not averse to formal 
assessment, thus warning against broad assumptions: 

... Many students expect/desire formal assessment tasks, others don’t. 

Especially low-level students. Some students like tests and relish their improvements. 

... my experience is most enjoy these indicators of success and progress. 

I am not convinced that most are because every individual wants to know where they are in 
the learning phase and that someone has the expertise to help them. 

Depends on the group. Most community students don’t want assessment. Workplace stu- 
dents want assessment for extrinsic, e.g. financial, reasons. 

As an adjunct to Statement E, Statement F presents informal assessment as a less stressful mode. 
Again, a strong endorsement of informal methods is displayed - 80% agreeing that most adult lit- 
eracy students are not intimidated by informal assessment. One comment showed some practical 
misgivings: 

This (informal assessment) is best assessment practice but hard to record and report. 

Other comments showed that informal assessment is one strategy amongst others: 

\ 

This is where a variety of assessment methods is a must. One, so you can ensure no student 
is intimidated and two, to ensure that the facilitator can assist all students when it comes to 
coping with different assessment strategies. 

... the implication is that we are comparing two methods here, formal and informal - (they) 
can be used successfully in concert. 

There is still the question as to how valid the teachers’ impressions of student reactions are, 
though these impressions would presumably be based on experience. However, the defence on 
behalf of the students’ perceived reception of formal assessment is the commonly expressed 
rationale behind practitioner reluctance to employ formal methods. This seems to be the crux of 
the arguments against formal assessment methods, and warrants further research to ascertain the 
nature of student reactions, from the students’ point of view. 

Statement G shows that, for a majority (78%) of the respondents, assessment is not an after- 
thought, but rather considered to be an integral part of adult literacy program planning: 
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Even informal or self-assessment techniques need to be thought about, planned etc. 
Otherwise we have no goals. 

There was a weak association suggesting that the more specific training the respondent had 
had in adult literacy assessment, the more positive their attitude towards assessment as an 
ongoing program feature. There was no association with the degree of practitioner’s adult 
literacy experience. Additional comments illustrate that planning is often balanced with 
flexible adaptation as the program progresses: 

... Sometimes it is necessary to change assessment strategies mid-stream to ensure all 
students are catered for. 

May need to be modified, but a forward plan is important. 

Statement H suggests that assessment processes should be standardised. A slim majority 
(52%) disagreed with this idea, many disagreeing strongly. There was no significant rela- 
tionship between this attitude and other respondent characteristics, though a weaker trend 
suggesting that full-time workers were more negative than others. Comments in opposition 
of standardised assessment tools pointed out the contextual nature of literacy practices: 

If we believe that literacy is a highly contextual, variable social practice, how can we 
adopt this stance? 

and the need to cater for a variety of learning (and appropriate assessment) styles amongst 
the student group: 

Guides are helpful but each student has their own set of unique problems and assess- 
ment tools can become too rigid to ascertain what a person learns. 

The comments revealed a strong sense of different practices suiting different levels of stu- 
dent competency: 

Okay for level 3 and above ... Not for Level 1 and 2 - very individual. 

Those agreeing with standardised assessment instruments saw their value more as item 
banks to be adapted for local usage: 

Curriculum should provide models of assessment tasks which can be contextualised 
for students at different sites. 

I don’t think everyone should be assessed the same way, but standard assessment 
tasks with marking guides would be a useful resource for teachers. 

For some practitioners, standardised assessment would obviously take some of the angst out 
of fulfilling their assessment duties: 

At least we could say everyone has to do them! 
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Respondents were finally asked to provide a general statement outlining their personal attitude to 
assessment practices in adult literacy provision. Of the 32 responses received, seven statements 
were strongly opposed to the new focus on formal assessment methods, for example: 

It sucks. I do believe that adult literacy practitioners do need to develop more defined con- 
cepts of sound theory and best practice BUT dancing to the tune of systems which require 
us to place our students in anti-learning environments will not facilitate this process. 
Assessment in adult literacy programs should only be self-assessment by the students. 
Program effectiveness needs to be evaluated on how well we facilitated the students’ 
achieving their own needs and goals. 

In 1 1 statements, the disadvantages of formal assessment practices were described. These 
included: 

• the stress placed on students; 

... formalised assessment scares the living daylights out of my students. 



• the lack of flexibility in formal assessment regimes to accommodate individual needs; 

When dealing with adult literacy students, their level of competencies vary so widely, it 
becomes extremely difficult to create ONE assessment task which will meet the needs of all 
participants. 

Formal assessment and reporting requirements tend to have us push students into “boxes” - 
with all having to do the same ... formal assessment. 

• the purpose to which such assessments are put; 

The difficulty with formal assessment is always going to be the interpretation of the assess- 
ment by the agency or organisation receiving it, for their purposes and needs. 

At a recent meeting our institute director stated the likelihood that we would be funded in 
the future on module outcomes. Obviously this would be a joke in the adult literacy area. 
This does however appear to be an option and direction that assessment and accountability 
seem to be heading. 

• and the negative reception of assessment requirements by volunteer tutors involved in pro 
vision; 

The requirements of structured courses, formal assessment and reporting has caused a great 
deal of concern from the volunteer tutors ... The tutors are our greatest resource and if we 
try to bring in formal structured courses for them to follow we will lose a large number of 
them. 

Volunteer tutors (without whom I could not function) complain that it is not so much 
‘fun’ any more. The formality of the course and required assessments is sometimes sti- 
fling. 
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Six statements tied the issue of assessment as an adjunct to their view of overly-prescriptive 
CBT curricula, that restrict the practitioner’s flexibility to respond to individual student 
needs; 

My fear is that these requirements will become a rigid set of rules, that that will force 
what is to be learned, and how it is to be learned, on every adult learner... I would not 
like to see some rigid set of rules that will destroy a learner ‘s right to achieve in their 
own choice of learning knowledge and their learning style and rate of learning. 

... a pre -determined outcome and assessment process is too inflexible and inappropri- 
ate to the learner’s needs. 

and to respond to special target group needs; 

The curriculum is not always appropriate. Aboriginal students require adaptable, flexi- 
ble programs with a cultural emphasis. Students with intellectual disabilities cannot 
be assessed in a standardised and formal way. They often require special programs. 

For three respondents, assessment against the curriculum’s explicit learning outcomes has 
become an end in itself: 

I see assessment driving some programs. My belief is that it is given too much 
emphasis as it is informal assessment which has the most potential to improve teach- 
ing and learning. 

More specifically, some respondents described philosophical problems with the assess- 
ment/reporting tools such as profile instruments, that they were required to use: 

I teach something, assess it, and the student is marked T = ‘competent’. Yet, some- 
times I know full well that if I were to repeat the assessment next week, the result 
might be very different. People with learning problems don’t fit the mould like most 
people. Even though they are making steady progress, they are not always consistent 
or predictable. Depending on the day, the mood and other variables, formal assess- 
ment can be either unfair or falsely generous. 

I have concerns with judgements, how misleading a judgement may be and how 
impossibly reductionist and absolute numerical scores can be - e.g. ILCM levels. 

The quotes above illustrate the nature of practitioners’ resistance to formalising assess- 
ment strategies in adult literacy. However, seven statements recognised the advantages of 
the shift in practice: 

Formal assessment is necessary and desirable. Students need to be able to measure 
their progress and claim awards. It also leads to more rigorous planning and more 
structured classes. It facilitates moderation and promotes standards. 

... assessment is necessary to prove achievement of objectives. It gives the course 
credibility. 

and the fact that some students will derive satisfaction and motivation from participating in 
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formal assessments: 

Some students are motivated more. Formal assessment practice prepares students for fur- 
ther study and education and very often educates in discipline necessary for entry to a 
workplace. 

Formal assessment gives our clients a goal to work towards... . 

The strongest statement in favour of embracing the wider agendas of competency-based accred- 
ited provision and more rigorous assessment regimes, highlighted the gains made by the field 
through its participation in wider agendas: 

It is time for ‘literacy* to come in from the backblocks, to be recognised as providing the 
building blocks of training, education and social practice, rather than an appendage to main- 
stream courses. Adherence to the ‘warm and fuzzy* image will eventually lose the field the 
gains it has made in recent years. The ‘language, literacy and numeracy’ field of training 
and education must demonstrate to industry, schools etc. that a rigorous programs of 
lang/lit/num provides a solid foundation from which to develop or with which to develop 
‘mainstream’ courses and industry training. 

The majority of other statements that recognised advantages and disadvantages in more formal 
assessment practices, expressed comfort with a compromise situation whereby systemic assess- 
ment requirements reflected the level of the students involved. Informal assessment was consid- 
ered appropriate for lower level students, and more formal practices appropriate for higher lev- 
els, with options for students to elect to participate or not in the assessment tasks. A total of 1 1 
responses suggested they would be comfortable with such a tiered system. The following quote 
is one example of the reasoning given: 

There is a need for assessment in adult literacy but I think formal assessment is only neces- 
sary at levels 3 and 4 (of CNL03), when the student knows that a satisfactory standard is 
necessary in order to receive the Certificate in Vocational Access. At the lower levels 
when the student is struggling and probably will always do so, formal assessment is another 
barrier to their beginning to feel some sense of achievement and stops them from continu- 
ing their learning. Many of these students are progressing sideways rather than longitudi- 
nally and there is no formal assessment for this, although both student and teacher know 
when progress is being made. It is only when levels 1 & 2 are assessed formally that they 
lose heart and begin to discount progress in other areas. 

Elsewhere, respondents articulated other qualities of what they considered desirable models of 
assessment. A number of these comments demonstrate continued adherence to the traditional 
individualised, social welfare philosophy of adult literacy: 

Assessment should be student-centred, in context and relevant to the student’s needs. 

I prefer to fit the learning outcomes to meet individual student goals and use informal 
assessment to measure competence. 

The assessment of a student wanting to do, or having done, a literacy course should be 
largely about confidence, attitude, self-esteem, independence etc and little about levels and 
grades. 
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Others argue that qualitative and self-assessments should be acceptable modes of assess- 
ment-cum-reporting: 

I think the students’ observations of how well they are doing should be sought as well 
as the teacher’s ongoing record of progress. 

The most important aspect of assessment is to give the student realistic feedback on 
their progress. Therefore assessment must reflect their goals and aspirations and be 
tied to the course work they have been doing. ... We must go beyond “X is making 
good progress” and be able to articulate what it is that X has achieved. 

As teachers we need to develop assessment practices that he somewhere between 
(CBT) and rubber band assessments. 

This spread of attitudes suggests that, as a professional body, adult literacy practitioners 
are in the process of being converted to an acceptance of more formal assessment practices. 
However, a major foundering point in their transition seems to be the compromise of flexi- 
bility to tailor provision and assessment to a client group, in favour of explicit, prescrip- 
tive curriculum that drives the assessment process. This flexibility was the traditional 
strength of adult literacy provision, and the field, it seems, would rather explore ways and 
means of accommodating such flexibility within the emerging agendas, than sacrifice it 
entirely. 

4. DISCUSSION 

External agendas such as the introduction of competency-based tr ainin g, accredited curricu- 
la and external accountability requirements have changed the face of literacy provision, and 
its assessment practices in particular. The changes from student-centred programs 
informed by invisible, non-intrusive informal assessment practices, to curriculum-driven 
programs incorporating “high-stake” assessment by visible, formal methods to report to 
third parties, have been imposed from above. As a result, the field has had to adopt prac- 
tices it previously eschewed or dismissed as irrelevant to the field, creating potential ten- 
sion for the practitioner between the “confronting discourses” of inherent attitudes and 
imposed practices. The transition period has seen a growing employment of profile report- 
ing instruments, increased use of formal assessment practices, and various efforts to provide 
professional development to practitioners involved in the transition. This study attempted 
to survey assessment practices across literacy sectors and to document the underlying atti- 
tudes that support or conflict with these practices. 

The practitioners sampled were in general mature, qualified teachers with a breadth of edu- 
cational experience. The level of specific training in adult literacy assessment emerged as 
the most influential factor in distinguishing patterns in assessment practices and attitudes. 
The bulk of respondents had only had minimal to moderate levels of training in assessment. 
Those with higher specialist qualifications had more specific assessment training to draw 
on, and this showed in their higher levels of confidence to undertake their assessment 
duties. Tutor scheme coordinators and tutors had typically low levels of training in assess- 
ment, which suggests that this sector will need additional developmental work before they 
can participate confidently in wider assessment /reporting requirements. Respondents who 
had a moderate level of assessment training were not convinced that their training equipped 
them for their assessment duties as delineated in explicit curricula. 
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A new constellation of program characteristics has emerged in response to the external agendas, 
in particular in the TAFE sector, constituting a new culture of literacy provision. These include 
the use of explicit, accredited CBT curricula, reporting requirements using profile instruments, 
specialist qualifications and professional development opportunities. These characteristics present 
as an interdependent package of reforms. Increasingly formal assessment practices that are sub- 
ject to scrutiny seem to be part of the package that the practitioners cannot avoid. This begs the 
question whether the field would have chosen the path of more formalised assessment, without 
the wider systemic changes. As this culture develops, the gap between this accredited sector and 
others will grow, making it difficult to introduce blanket strategies, such as the NRS. Not all 
providers will be at the same starting line. 

This study has shown that literacy practitioners are employing a wide range of assessment strate- 
gies, both formal and informal, and consider assessment an important part of overall program 
planning. Each practitioner has a repertoire of strategies, not just relying on one or two. The tra- 
ditional preference for informal, incidental assessment has not been discarded, but rather practi- 
tioners are accruing more strategies into their repertoire to complement this one. In addition 
they expressed interest in acquiring more, with the assistance of focussed professional develop- 
ment and support materials such as item banks. Most assessment practices were evenly spread 
across the sample sectors, apart from formal formative and summative practices. These latter 
strategies were utilised only in programs with explicit curricula, reporting requirements and 
qualified staff, underscoring the constellation of factors shaping a new culture of provision. 

Though participating in more formal assessment strategies, the respondents still did not value 
these practices highly, putting much more emphasis on the importance of informal assessment in 
their programs. They were not on the whole convinced about the possible benefits of formal 
assessment to the students, but they were convinced that formal assessment carried risks for their 
students. Specific training in adult literacy assessment seems to encourage more positive attitudes 
to formal practices. However, there remains a strong core of attitudinal resistance to formal 
methodologies among literacy providers, at the crux of which is the concern over the impact of 
formal assessment on students’ confidence, and the perceived loss of flexibility to tailor provision 
and assessment to individual needs. Though some respondents could identify students who will- 
ingly participated and gained from formal assessment practices, this protective, negative attitude 
persists. Further research is needed to substantiate or challenge this common perception of stu- 
dent attitudes. 

There is another cohort of respondents who in general could acknowledge advantages in the 
direction assessment practices were taking, but had some misgivings at this point. These con- 
cerns included their own lack of training to inform their new practice; reservations about some of 
the assessment tools or processes they were required to use; and the constant tension between 
meeting student needs in a prescriptive curriculum. For this group, a frequently expressed com- 
promise would be tiered or phased assessment strategies, whereby low level students were 
assessed informally, and more formal practices introduced as the student’s competency increased. 
There was some interest expressed in standardised assessment tasks for any curricula, but com- 
ments showed this to be largely an interest in the provision of support materials, models of good 
practice, and item banks to be adapted to local settings, rather than for tightly controlled assess- 
ment instruments. 

At the other end of the scale, the sample included some strong advocates for the move to more 
visible, rigorous assessment practices, frustrated by the field’s continued adherence to ‘"warm, 
fuzzy”, “rubber band” practices. These practitioners had made the transition between paradigms 
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and were willing participants in the wider agendas at play. 

In summary, adult literacy is a field in a state of transition, with sectors and individuals at 
various stages of transition towards the emerging culture of accredited provision. The 
imposition of directions from above or from outside the field has produced tensions 
between attitudes and required practices, but this study suggests that attitudes are gradually 
shifting in the wake of changing practices. If provider organisations are prepared to listen 
to practitioner’s suggestions on how to adapt or support assessment practices, the result 
could accommodate strengths from interests both internal and external to the field. 
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Appendix 1:1 
The Questionnaire 



15 March 1996 
Dear 

What do you think about assessment in adult literacy and numeracy programs? 

I would like to invite you to respond to the enclosed survey about your current assess- 
ment practices and attitudes to assessment in adult literacy and numeracy programs. 

This survey is an action research project funded by the National Languages and 
Literacy Institute of Australia (NLLIA). It arose through a concern that with develop- 
ments such as the National Reporting System to be implemented in 1997, all literacy 
providers have to increasingly participate in structured assessment and reporting processes. 
This is in response to an increased demand for accountability to funding bodies on tangible 
“outcomes” of literacy programs. This shift marks a shift from the earlier philosophy of 
adult literacy provision that tended to avoid formal assessment in the interests of building 
students’ confidence. This survey is an attempt to gauge how practitioners feel about such 
developments, and how confident they are in carrying out their evolving assessment duties. 

I know you are probably busy, but your response would be very valuable. Hopefully the 
results of the project will inform future policy and professional development strategies in 
the area of assessment practices. 

Your identity will remain confidential, and at no stage will your responses be used in such a 
way that could identify you or your employer. If you have any concerns or queries, please 
contact Cathie Doherty, on (076) 38 3346. 

Please return the completed survey form in the envelope provided, by Friday 29 March. 
Your contribution will be most appreciated. 

Yours sincerely 



CATHIE DOHERTY 
PROJECT OFFICER 
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THE NATIONAL LANGUAGES AND LITERACY INSTITUTE OF AUSTRALIA 
QUEENSLAND ADULT LITERACY RESEARCH NETWORK NODE 



THE IMPACT OF ASSESSMENT AS AN ONGOING FEATURE 
OF THE LEARNING ENVIRONMENT 



CONFIDENTIAL 



ANY QUESTIONS OR PROBLEMS? 

PLEASE CONTACT CATHIE DOHERTY ON (076) 38 3346 



PLEASE RETURN IN THE SUPPLIED ENVELOPE TO: 

DOROTHY LLOYD 
TOOWOOMBA COLLEGE OF TAFE 
P0 BOX 80 
TOOWOOMBA 4350 

BY FRIDAY 29 MARCH. 
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Appendix 1:3 



SOME DEFINITIONS: 

For the purposes of this study, please consider the following terms to mean: 

Assessment: 

Any process or procedure whereby judgements are formed of a student’s degree of compe- 
tence, strengths or weaknesses, or of his/her progress towards the objectives of a learning 
program. This includes formal and informal assessment techniques. 



Formal assessment: 

* assessment that is planned and undertaken consciously by the teacher; the outcomes 
of which are systematically recorded, to inform any reporting requirements; 

* the same assessment is applied to all students in the program, under similar conditions; 

* students are aware that they are being assessed. 

This could include test/exam procedures, systematically collecting samples of work, or 

targeted and recorded observations of students’ performance in learning tasks. 

Informal assessment: 

* ongoing cumulative judgements and feedback on student progress derived from 
incidental observation and participation in the learning activities of the program; 

* outcomes may not be recorded, but add to the teacher’s incremental mental picture of 
the student’s performance; 

* students may not be aware that judgements are being made; 

* assessment judgements are made in contextualised interactions, which may differ 
between students. 
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Please supply the following information. At no stage will vour responses he used in such a 
wav that could identify^vou orvour program. Where there is insufficient space, please 
attach an additional sheet. 

SECTION 1 - ABOUT YOU 

1 . I consider my community to be: 

( ) Predominantly metropolitan 
( ) Predominantly regional urban 
( ) Predominantly rural 

( ) Other, i.e. 

2. I am: 

( ) FEMALE 
( ) MALE 

and ( ) 25 years old or less 
() 26 - 35 years old 
() 36 - 45 years old 
( ) 46 years old or more 

3. I currrrently work in adult literacy as: 

( ) a part-time teacher 

( ) a full-time teacher 

( ) a volunteer tutor 

( ) a volunteer tutor coordinator 

( ) other, i.e. 

4. I work in: 

( ) a TAFE program 
( ) a Skillshare program 
( ) a community program 
( ) workplace programs 

( ) other. Please specify: 

5. a) Is your program based on a curriculum that states specific learning outcomes? 

() No - Go to Question 6 
( ) Yes. 

b) If Yes, Are you required to provide a report on each student at the end of the 
program against these learning outcomes? 

( ) No 
() Yes 

6. I have worked in adult literacy for: 

( ) 1 year or less 
( ) between 1 to 5 years- 
( ) more than 5 years 
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7. I have other teaching experience as follows: 

Years: 

Years: 

Years: 



Appendix 1:5 



8. I have the following educational qualifications: 

PRIMARY: 

SECONDARY: 

TERTIARY: 

OTHER 

9. I have had the following training or professional development in the specific area oTassesfe 
ment for adult literacy or numeracy : ( e.g . a semester unit on assessment in my teaching 
qualification; a 2 hour session in my volunteer tutor training ; as part of the Adult Literacy 
Teaching professional development program....) 

Description Approximate hours Year 

of instruction 



10. Prior to your experience in the field of adult literacy or numeracy, what training in assess- 
ment have you had? 

Description Approximate hours Year 

of instruction 
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SECTION 2: ABOUT YOUR ASSESSMENT PRACTICES 

11. In my adult literacy program/s, I incorporate the following assessment practices: 

(Tick any relevant response. Please provide additional information if your practices are not 

adequately described.) 

( ) a. Initial diagnostic assessment for placement and/or program pl annin g 

( ) b. Ongoing informal observation and consideration of my students’ progress, strengths 
and weaknesses. 

( ) c. Systematic recording of my observations about my students’ progress on an ongoing 
basis. 

( ) d. Regular tests of word recognition and phonic skills. 

( ) e. Maintain a portfolio of samples of students’ work to demonstrate their progress. 

( ) f. Encourage self-evaluation by the students throughout, the program. 

( ) g. Facilitate self-evaluation by students of their overall progress on completion of a 
course. 

( ) h. Regular tests on agreed list of number facts or words (for spelling or vocabulary). 

( ) i. Formally administered assessment tasks such as tests that reflect the program objec 
tives, as checkpoints through the course (formative assessment). 

( ) j. Formal assessment tasks at th e en d of the course to assess overall student progress 
against stated outcomes, facilitate future placement or to fulfil any reporting require 
ments (summative assessment). 

( ) k. Participate in moderation of students’ performance on assessment tasks with other lit 
eracy or numeracy teachers. 

( ) 1. Match my students’ performance on assessment tasks to some supplied descriptions 
or profiles of competency levels (e.g. ALAN scales, or the Interim Literacy Course 
Matrix/ILCM) 

( ) m.I contribute to other staffs assessment of my students’ progress. 

( ) n. OTHER. Please provide a brief description: 
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12. What assessment practices or procedures would you like to incorporate in the future? 
(Nominate items from Question 11 above , or describe briefly.) 



13. What additional training/expertise/resources/support would you consider necessary to 
implement these practices (nominated in Q.12) ? 



14. During your experience working in adult literacy, what changes have you witnessed in the 
area of assessment requirements and expectations? 
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SECTION 3: ABOUT YOUR ATTITUDES TO ASSESSMENT IN ADULT LITER- 
ACY 

NOTE: PLEASE BE ASSURED THAT YOUR COMMENTS WILL NOT AT ANY 
STAGE BE USED IN SUCH A WAY THAT COULD IDENTIFY YOU OR YOUR 
PROGRAM. 

QUESTIONS 15-22: Consider the following statements and indicate on the scale to 
what extent you agree/disagree with each statement, IN GENERAL. Feel free to supply 
additional comments where desired. 

15. Formal assessment forms an important and conscious part of my teaching strategy. 
STRONGLY STRONGLY 

AGREE DISAGREE 

5 4 3 2 1 



COMMENT: 



16. Informal assessment forms an important part of my teaching strategy. 

STRONGLY STRONGLY 

AGREE DISAGREE 

5 4 3 2 1 



COMMENT: 



17.1 feel my training has equipped me well to undertake the program’s required assessment 
duties. 

STRONGLY STRONGLY 

AGREE DISAGREE 

5 4 3 2 1 



COMMENT: 



Appendix 1:9 

18. Adult literacy students benefit from the clear indication of their levels and progress 
that formal assessment can give them. 

STRONGLY STRONGLY 

AGREE DISAGREE 

5 4 3 2 1 
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COMMENT: 



19. 



Most adult literacy students are intimidated and discouraged by formal, structured assess 
ment tasks, such as tests. 

STRONGLY STRONGLY 

AGREE DISAGREE 

5 4 3 2 1 



COMMENT: 



20. Most adult literacy students are not daunted or discouraged by informal assessment 
processes, e.g. interviews and classroom observations. 

STRONGLY STRONGLY 

AGREE DISAGREE 

5 4 3 2 1 



COMMENT: 



21 . It is important to develop an assessment strategy at the same time as developing program 
objectives. 

STRONGLY STRONGLY 

AGREE DISAGREE 

5 4 3 2 1 



COMMENT: 
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FILLING IN THE BLANKS: 

CONSTRUCT VALIDITY ISSUES IN THE 
DEVELOPMENT OF ASSESSMENT TASKS 
FOR ADULT LITERA CY STUDENTS 

Ann Kelly 



Introduction 

Wagner (1993) has suggested that when constructing knowledge in a disciplinary field, 
researchers work within either of the following two paradigms of ignorance. They fill in 
blank spots, that is, respond to questions already raised by other research colleagues and 
practitioners. Alternatively, they identify blind spots, that is, pose new questions for the 
field. The work reported in this chapter fits the first category, filling in the blanks with 
respect to the moderation of assessment processes in an adult literacy curriculum. In the 
discussion section, however, the wider question of the adoption of competency-based train- 
ing in adult literacy is addressed. 

In 1995 adult literacy teachers working within TAFE institutes in Queensland were required 
for the first time to provide programs that met competency-based training guidelines. The 
development and conduct of an assessment procedure are implicit in this requirement. Yet, 
systemically, very little formalised staff development is available to assist teachers with the 
resulting new assessment demands. 

In the adult literacy field, two initiatives were taken in 1995 by the Queensland Research 
Network Node of the National Languages and Literacy Institute of Australia (NLLIA) to 
improve this situation. During the first half of the year, Joy Cumming and Christa van 
Kraayenoord each presented a number of workshops focusing on assessment issues. The 
second initiative involved the offering of small grants to adult literacy teachers to enhance 
their assessment practices. Sharon Mullins, Ruth Kemp and I, teachers at the Southbank 
Institute of TAFE, received one of these grants. 

In our submission to the NLLIA committee we proposed to invite practitioners in TAFE 
institutes to provide assessment items related to the Personal Focus strand of the Certificate 
of Vocational Access course. This strand is one of four in the certificate course, the others 
being Employment, Vocational and Workplace. The personal domain was chosen for two 
reasons: our own experience in teaching this particular focus and the contentious nature of 
the suitability of a competency-based framework within the personal domain of people’s lives. 
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We also invited practitioners to provide examples of student responses to these assessment items. 
However, when it became obvious that very few examples of student work would be available, 
we decided to limit our collection to assessment items. The purpose of collecting these examples 
was to initiate a moderation process whereby these items would be validated by practitioners in 
the field. 

In this chapter the literature that informed our work and the process that we adopted as a first 
step towards the establishment of a bank of assessment tasks, assessment criteria and instructions 
or rubrics for completing the tasks in the personal strand are detailed. It was anticipated that the 
resultant material could then be used as models for assessment practice in the adult literacy field 
in Queensland. In a subsequent section the action leaming/action research process that was adopt- 
ed in the project is described. Finally, some reflections on wider considerations of competency- 
based training are presented. 

Adult literacy assessment and moderation conventions 

Until the implementation in 1995 of the competency-based curriculum, the Certificate of 
Vocational Access, assessment in the adult literacy field in Queensland was not prescriptive in 
either content or process. At best, assessment practices reflected the principles enunciated by the 
Australian Council for Adult Literacy in 1992. These are reproduced in Figure 1. These princi- 
ples had been published in a position paper in an attempt to provide guidelines for best practice 
in assessment and to ensure that performance scales were not used in adult literacy contexts 
(Griffin & Forwood, 1991). Generally, however, little formal assessment of adult literacy course 
delivery was performed in Queensland before 1995. This situation also prevailed across Australia 
until the early 1990s (Cope et al., 1994) and was consistent with student-centred, local, flexible 
approaches to provision that existed at the time. 



PRINCIPLES OF ADULT LITERACY AND BASIC EDUCATION ASSESSMENT 

1 . Assessment should be multi-dimensional, that is, tap attitudes, knowledge, processes and 
skills with multiple measures acknowledging the complex nature of literacy. 

2. Assessment tools and reporting mechanisms should be developed in conjunction with and 
be informed by curriculum frameworks, specific contexts and articulation processes. 

3. Assessment should be continuous and integral to the teaching/leaming process (and the cur- 
riculum). The process should be costed separately, i.e. identified in funding submissions. 

4. Assessment tools need to be based on consistent and appropriate theories which make 
explicit relationships between language and the socio-cultural context and use authentic and 
relevant texts and tasks. 

5. Assessment procedures (tools, etc) should be regularly reviewed (validity, reliability, appro- 
priate use). 

6. Assessment should be a collaborative process carried out in an environment conducive to 
allowing students maximum opportunity to demonstrate ability. The rights, needs, sensibili- 
ties and integrity of the individual are the prime consideration. 

7. The purpose of the assessment must be explicit and clear to all stakeholders: the student, 
the assessor and whoever commissions the assessment. 
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8. Reporting systems must be understandable and accessible and to some extent, stan- 
dardised. 

9. There should be different kinds of reporting mechanisms for different stakeholders. 

10. The duration of the assessment report should be specified. 

1 1 . Assessors should have appropriate training. 

12. The ‘accreditation’ or ‘registration’ of assessors should be supported. 

(Australian Council for Adult Literacy, 1992) 



Figure 1 

A similar lack of formality prevailed for the use of moderation conventions. The definition 
of moderation coined by Lyons (1994:21) guided the project team. Lyons’ suggested that 

moderation normally refers to the range of activities that are designed to ensure parity 
and fairness in assessment practice and decision - achieved by clarifying and develop- 
ing common understandings of standards and learning goals and enhancing validity 
and reliability in assessment design and application. 

Because moderation processes are linked closely to assessment practices, it is not surprising 
that there is no past evidence of national or systemic unanimity in conceptions of best prac- 
tice in moderation within the adult literacy sector. In fact, there is virtually no mention of 
moderation in the adult literacy literature until this decade. 

Assessment and moderation within a competency-based framework 

Assessment issues are an integral part of the literature relating to the implementation of 
competency-based training in Australia. However, two approaches to the issue of assess- 
ment issues are discernible. One form has been through the development of procedural texts 
(Gonczi, Hagar & Anthanasou, 1993; Hagar, Anthanasou & Gonczi, 1994) to accompany 
the Vocational Education, Employment and Training Advisory Council’s (V EE TAC, 1993) 
competency standards framework. VEETAC has been positioned as the body charged with 
overseeing key competency-based policy developments in Australia. VEETAC’s publica- 
tions have been drawn upon in this chapter to contextualise the assessment principles that 
have been recommended for use within the vocational sector. The second form is a substan- 
tial corpus of critique. The authors contributing to this body of work include Biggs (1994), 
Burrows (1993), Hall (1995/6) and Wolf (1995). While these co-exist in the research com- 
munity, there is virtually no promotion of substantive critique evident within vocational 
training institutions (Stevenson, 1992). Through this critical work, the sanctioned model 
guiding competency-based assessment will be examined and an alternative assessment 
model will be presented. 

In this section the model of assessment developed by VEETAC (1993) is presented and the 
implications of this model for use by adult literacy teachers in the Queensland sector will 
also be discussed. 
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In 1993, assessment was defined by VEETAC (p.13) as: 

the process of collecting evidence and making judgements on the nature and extent of 
progress towards the performance requirements set out in a standard, a learning outcome 
and, at the appropriate point, making the judgement on whether competency has been 
achieved. 

Four essential features of competency-based assessment were identified in this framework: valid- 
ity, reliability, flexibility and fairness. However, VEETAC provided little attending elaboration 
about how these features might be addressed by teachers in developing assessment processes. To 
remedy this situation, Hagar, Anthanasou and Gonczi (1994) produced two documents, 
Assessment: Technical Manual and its companion, Assessment: Practical Guide. The validity 
aspects of the technical manual were used initially to provide a focus for our study. Validity is 
generally accepted to mean that assessment measures actually test or judge what they are intend- 
ed to test or judge. The prime positioning of validity and the relegation of the other features of 
the framework to sub-set positions within the validity construct are consistent with Linn’s (1994) 
view. 

Linn asserts that considerations such as reliability and fairness are only important to the degree 
that they assist in ensuring that assessment measures are valid or indicate that action needs to be 
taken to improve the degree of validity of the measures. 

The following principles frame the concept of validity within the VEETAC (1993) document. 

* The assessment criteria must be sufficiendy broad to encompass the range of skills and 
knowledge necessary to demonstrate specific competencies. 

* The assessment methods and tasks must be capable of determining whether assessees pos- 
sess the cognitive attributes that underpin these competencies or not. 

* Sufficient evidence must be provided on a range of occasions and across diverse contexts to 
ensure that valid judgements can be made. 

* Monitoring and review processes should be established to ensure that both task selection, 
assessment criteria and rubrics are interpreted in consistent ways. 

They are discussed separately in the following section. 

Breadth 

The issue of breadth of coverage is critical in the assessment of competence. Yet it was not 
always considered in the development of the Certificate of Vocational Access. In the module LIT 
113, Literacy and Language Personal Focus, Level. 2, the learning outcome is “to use literacy and 
language texts related to participation, discussion and decision making within groups within the 
community”. Two tasks are suggested as exemplars: read and act on minutes of meetings, and 
take personal notes. These tasks are fine. However, it is noticeable that the assessment criteria 
relate only to participation in a meeting. This specificity is too narrow. Assessment criteria 
should be sufficiently broad to apply to a range of tasks that would be appropriate for this learn- 
ing outcome. 

To be fair, this problem is not common in the Certificate of Vocational Access and is likely to be 
remedied in the draft to be available in 1996. 
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Underlying knowledge and skills 

The second principle in the VEETAC framework focuses on the assessment of the under- 
standing and skills inherent in the performance of tasks. This is not a simple matter. Soucek 
(1993:166) suggests that underlying knowledge and skills might be conceptualised as con- 
stituting three domains: “linguistic, interpersonal and cognitive”. Even this construct may 
not be sufficiently elaborate. For example, Stevenson (1993a), drawing on the work of cog- 
nitive psychologists such as Brown, Scardimalia and Collins, differentiates between specific 
and higher order processing skills. Further, Soucek (1993) claims that, while the assessment 
of understanding and skills might provide an indication of performance, the reverse does 
not hold. For example, a person may be able to apply a particular algorithm to solve a 
mathematical problem but he or she may not understand whether that algorithm is an appro- 
priate one. 

In order to be in a position to assess the knowledge and skills related to specific competen- 
cies, two steps must be taken. Firstly, the specific knowledge underpinning the skill must be 
identified. This must then be translated into assessment criteria and task instructions. An 
example showing slippage between the assessment criteria and task demands can be found 
in the second learning outcome in LIT 1 1 1 Literacy and Language Personal Focus, Level 1 
A of the Certificate of Vocational Access. It states that competency is achieved when an 
assessee can ‘f ead and record very short, explicit, context-embedded texts associated with 
membership of and interaction within a relevant group ”. A suggested task is for assessees to 
prepare a list of names (minimum 5) of significant members of a relevant group in order to 
complete a relevant task. The following assessment criteria are stipulated: 

• write the names from memory 

• write the names so that they can be understood 

• explain the list orally and carry out a task using the list 

• explain how the spelling of names will be checked before writing invitations, cards, etc. 

In the exemplars from our own classes and those provided by colleagues that we examined, 
we found that different interpretations were evident. In one case, students were asked to 
“Write the names of four of your family members”. There are constructs here that are being 
assessed without explication. In relation to the construct “name”, what constitutes a name? 
Is a first name, first and family names, or any other possible combinations, required? 
Additionally, the intention of the assessor in relation to the construct of “family members” 
is not clear. In a multi-cultural society such as ours the word ‘family” can be interpreted 
variously. 

A similar example provided by another assessor was more explicit. Student were required 
to: “Write a list of 5 people in our class. Beside their names, write a food item for them to 
bring to a class morning tea. Use their first names only.” While the instructions could obvi- 
ously be written more logically, this example meets the outcome criteria well. It was 
accompanied by a clear set of instructions and an assessment criteria proforma. 

Sufficient evidence 

The general assessment literature is replete with exhortations to use a range of tasks to 
improve the degree of validity (Linn, 1994; Messick, 1994; Shavelson, Baxter & Pine, 

1992; International Reading Association/National Centre for the Teaching of English, 

1994). Similarly, there is a recommendation to this effect within the adult field. Principle 6 
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of the report, Assessment and Moderation Processes in Adult Literacy and Adult ESL inTendered 
Labour Market Programs by Cope et al. (1994:85) states: 

Multiple assessment tasks, and where possible several assessors, are used to assist in the 

reliability of the assessment outcome. 

This principle of usage of a range of tasks was critical in our project. We sought to moderate the 
range of assessment tasks, criteria and rubrics relating to the personal focus strand of the 
Certificate of Vocational Access so that a bank of tasks that are acceptable to the field are avail- 
able in the future. At this stage in the development of moderation processes within the TAFE 
system comparability of tasks and reliability between raters and across contexts and situations 
were not being addressed. 

Significant issues, hinging on interpretations of the constructs “sufficiency” and “evidence” in 
this principle, needed to be addressed in our project. “Sufficiency” could refer to the number of 
performances or the completeness of the assessment criteria to meet the outcome. There also 
needs to be a consideration of what breadth of evidence might constitute sufficiency. This aspect 
has been discussed earlier in this article. When the bank of tasks has been established, then fur- 
ther work will need to be done in responding to the sufficiency issue. 

The construct “evidence” is no less problematic. Hagar, Anthanasou and Gonczi (1994) suggest 
that the forms this might take include project and assignments, problems, case studies and written 
tests, simulations, skills tests, direct observations in a range of situations, and documentation in 
the form of portfolios, log books, qualification certificates references, supervisors’ reports. The 
Certificate of Vocational Access curriculum recommends that a range of texts from within the 
factual, persuasive, transactional, procedural, narrative and conversation range as they relate to 
the students’ goals and interests should be used for instructional purposes. The recommended 
assessment methods are equally comprehensive. They include practical tasks, written tasks, short 
oral explanations, short oral answers, simulations, role plays and scenarios. No mention is made 
however, how these might be equated either individually or in sets for validity purpose. 

Moderation and review processes 

It might be assumed that if there is in place an agreed selection of tasks and accompanying 
explicit assessment criteria and rubrics, then as long as assessors follow these rubrics and judge 
competence according to the criteria, then validity conditions will be met. This has not been 
shown to be the case. Wolf (1991) provides two telling examples in which assessors were shown 
to be higfily unreliable both between themselves and dining remarking processes, even after 
training. Along with others such as Harris, Lundberg, Hobart and Guthrie (1995), she warns that 
we must accept that “perfect transparency” (p.17) cannot be obtained in any competency-based 
assessment process. In addition to ensuring that the assessment criteria are as clear as possible, 
three other conditions should apply. Exemplars should be provided, training should be available 
to assessors, and marking systems should be monitored at regular intervals. Because of the need 
for cross-crediting at the upper levels of the Certificate of Vocational Access curriculum, it will 
be necessary to implement Wolfs recommendation at a systemic level. 

Cope et al. (1994) support the view that moderation is at the centre of good assessment practices 
and provide a number of assessment principles that they urge should be accepted by the field. 
These have been collapsed into the following three principles. 
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• There is evidence that all providers have a formalised moderation procedure in place 
and that teachers have access to banks of moderated assessment tasks and criteria. 

• There is evidence that all adult literacy teachers participate in at least a local level of 
moderation. 

• Moderation structures exist across providers. 

The systemic adoption of these principles should be taken up. However, we must be con- 
scious of Linn’s (1994) warning that such a system can develop into a bureaucratic control 
measure. Practitioners themselves should be involved in this work which could take the 
form of action learning processes. 

Assessment -within a performance-based framework 

While the VEETAC framework provided valuable guidance in our reflection on best prac- 
tice in assessment, we found we preferred an alternative model that was developed by Linn, 
Baker and Dunbar (1991) to inform performance-based outcome practice in the United 
States. This framework proposed that the following set of eight criteria be used as a basis 
for evaluating assessment. 

Consequences 

Linn, Baker and Dunbar (1991) emphasise the centrality of a consideration of the intended 
and unintended effects of assessment on those who have a stake in the process, primarily, 
teachers and students. One result of formalised assessment processes is the improbable 
washback effect on teaching and learning. Smith, a Director in TAFE in Queensland 
(1994:8), claims that “CBT provides a major window of opportunity for vocational educa- 
tors to position assessment as a truly integral part of teaching and learning”. This may, or 
may not occur. Improved delivery was an intended outcome of performance testing in the 
United States but research has shown that this was not always realised (Guskey, 1994). 

The issue of consequence, however, affects groups other than teachers and learners. Those 
with a particular interest in assessment results might include course administrators, employ- 
ers, job placement officers, the Australian National Training Authority or the Department of 
Employment, Education and Training, Commonwealth Employment Services or private 
placement officers. Each of these may have somewhat differing needs of assessment sys- 
tems and may place varying constraints on curricula decisions. Catering to these diverse 
demands is not easy. 

Moss (1994:6) suggests that a more comprehensive definition of validity ought to stress the 
value of “balancing differing concerns” as long as “acceptable levels are achieved for par- 
ticular purposes of assessment”. A similar position, defined as “a process of risk manage- 
ment” is adopted by Harris, Guthrie, Hobart and Lundberg (1995:162). Taking the exam- 
ples of an airline pilot and a doctor, where credentials confer high status and assume high 
levels of responsibility, the authors suggest that there are some contexts in which assess- 
ment processes should be more rigorous than others. Thus, the “consequences” issue must 
be addressed in moderation sessions, particularly when considering the upper levels of the 
curriculum where credit transfers to other courses operate. 



The criteria of fairness is one of the major criteria delineated within the VEETAC ffame- 



Fairness 
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work (1993). It is interesting to note that it is not included within a validity category in this work. 
There are four principles relating to fairness in the Linn, Baker and Dunbar (1991) model: 

• Assessment practices and methods must be impartial. No specific groups should be disad- 
vantaged. 

• Assessment procedures and criteria must be transparent to all potential assessees. 

• A participatory approach should be employed with process jointly developed and/or agreed 
between assessors and assessees. 

• Challenges to assessment processes must be available. 

However, in any consideration of fairness, there is an assumption that society is value-free. This 
is obviously not so. Linn, Baker and Dunbar (1991) cite a range of evidence to support the pres- 
ence of bias in results because of differences in task familiarity, exposure and motivation 
amongst different groups as well as rater responses in the United States. It is too early to know 
whether such differences exist in relation to the Certificate in Vocational Access curriculum but 
they are likely and will need to be addressed. 

Transferability and generalisability 

This third criterion in Linn, Baker and Dunbar’s framework has been of particular interest to 
Billett, a vocational education and training researcher at Griffith University. Following on the 
earlier work by Green (1989) and other cognitive psychologists, Billett (1994) proposes a number 
of features in learning settings that contribute to generalisations and transfer to other settings. 
These are: the learning experiences should match those performed in the real world as directly as 
possible; the total range of these experiences should be presented; and the knowledge and the 
skills underlying the experiences should be made explicit and practised to the extent that students 
can use effective problem-solving strategies to deal with related tasks. 

This list has implications for both teachers and assessors. Adult literacy teachers have been 
involved in providing courses within a range of workplace settings and are now developing 
expertise in understanding the discourses within them. However, this is not the case for commu- 
nity settings, particularly those of the students who are likely to be enrolled in the Personal Focus 
strand of the Certificate of Vocational Access curriculum. Currently, we are in danger of writing 
into curricula outcomes, tasks and assessment criteria that reflect our own values and understand- 
ings, rather than those of our potential students. The recent studies by Bull and Anstey (1994), 
Breen et al. (1994) and Freebody and Ludwig (1995) where they documented a range of literacy 
practices across groups and settings will assist in developing this knowledge, but more needs to 
be done. However, even when we can choose appropriate activities, the selection of criteria that 
taps the underlying knowledge and skills, including metacognitive ones, will not be easy. Finally 
the sampling of these tasks into constellations that ensure similarity or equivalence, as noted 
above, will continue to be problematic. 

Cognitive complexity 

The concept of task complexity is one of the major features underpinning the Certificate of 
Vocational Access curriculum. For example, the fifth learning outcome in the module LIT111, 
Literacy and Language Personal Focus Level 1 A requires competent assessees to be able to “use 
short, explicit, pictorial/symbolic language, literacy and numeracy texts related to commonly 
used procedures for completing relevant tasks to meet the learner’s individual goals”. In contrast, 
the fifth learning outcome of the module, LIT1 15, Level 3B of the same focus, the second high- 
est in the certificate, requires competent assessees to be able to “use literacy and language texts 
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of extended length concerning unfamiliar procedures that are related to tasks relevant to 
learner’s goals”. The constructs that are pivotal to determining complexity in the curriculum 
are length of text, transparency, familiarity, graphic was presentation and regularity of use. 
These same constructs should be assessed through the specific criteria developed to deter- 
mine competence on the various outcomes. 

Content quality 

While the Certificate of Vocational Access curriculum has a rich theoretical foundation, as 
evidenced by the references to the document, only two theoretical impositions are made 
explicit within the work. These are: a functional language model which aims to describe 
language in use (Rothery, 1 992); and a socio-cognitive model based on a particular text 
defining the role of the four constitutive roles of readers proposed by Freebody and Luke 
(1990). It is not easy for many practitioners, particularly those in remote regions, to access 
reference material. As a result, future curricula will need to state explicitly within the text 
itself the research base of our work in adult literacy. This extended, detailed research sup- 
port will also assist in the defence of competency decisions (Cope et al., 1994), if and when 
they arise. 

Content coverage 

This criterion has already been addressed by discussion of the earlier criteria within the 
VEETAC Framework document relating to breadth of assessment processes. 

Meaningfulness 

As Linn, Baker and Dunbar (1991:20) claim, “like motherhood and apple pie”, meaningful- 
ness should have a place in a set of criteria for evaluating assessment. The authors interpret 
the concept of “meaningfulness” in terms of authenticity of tasks. This factor is central to 
the adoption of both competency-based systems in Australia (Billett, 1994) and perfor- 
mance-outcomes in the United States (Messick, 1994). Meaningfulness can also apply to 
the relationship between specific outcomes and their related requirements to demonstrate 
competence in this outcome. If these are seen to be closely related, then it is likely that stu- 
dents will remain interested in participating in assessment processes and leam from the 
assessment experience. 

Cost and efficiency 

Economic and efficiency considerations, the final criteria in Linn, Baker and Dunbar’s 
(1991) list, are leitmotifs that continue to recur throughout vocational tr ainin g policy docu- 
ments. However, they must be counterpoised by other factors such as ensuring that there 
are multiple opportunities for students to demonstrate competence and establishing systemic 
monitoring procedures to ensure that assessment processes are valid. These steps will con- 
tribute to effectiveness and ultimately efficiency, but they will come at a price. 

Draft criteria for evaluating assessment tasks, criteria and rubrics 

Because of its comprehensiveness and applicability, the model by Linn et al., comprising 
the eight criteria discussed above, will be used as a guide for further work in the modera- 
tion of the assessment tasks, assessment criteria and rubrics provided by adult literacy 
teachers for this purpose. 
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Specifically, the following questions, in their current draft form, will be used by adult literacy 
teachers to determine the appropriateness of the assessment examples in future moderation work. 

• Is the context for the task stated? Is the task requirement stated clearly? Are the assessment 
criteria stated clearly? 

• Is the learning outcome addressed sufficiently by the type of assessment task chosen? Is the 
learning outcome tested both as a whole and through its components by the assessment cri- 
teria chosen? 

• Does the level of complexity of the task and the assessment criteria match the level inherent 
in the outcome statement? 

• Are the assessment task and criteria as authentic as possible? 

• Are the task and criteria likely to be meaningful to the assessee? 

• Is this task cost effective and efficient? 

• Does the criteria address knowledge and skills that could be generalised to another context? 
Action leaming/action research process 

The brief that guided our proposal for this study stipulated that an action research process would 
be adopted. Action research methods have become popular with teachers in Australia, largely due 
to the work of Kemmis and his colleagues (Kemmis & McTaggart, 1988) over the past fifteen 
years. In contrast, within the vocational education sector, action learning has been actively pro- 
moted. For example, substantial funding has been, and continues to be available through the 
National Staff Development Committee to encourage employees to engage in action learning 
processes to improve their practices. 

In this study, we attempted to tease out the differences between the concepts “action research” 
and “action learning” in order to situate our work. There are a number of common features. In 
both approaches, participants work collaboratively and reflectively on an issue or issues that orig- 
inate within their own social environments with the purpose of enacting change (McNiff, 1 992). 
However, while people primarily engage in forms of action learning to develop their personal or 
professional competencies, action researchers take this process one step further. Ballantyne, 

Bruce and Packer (1993) differentiate between action learning and action research by claiming 
that “action research tends to be more deliberate, systematic and rigorous than action learning 
and places greater emphasis on the evaluation and reporting of results”. 

Taking this model, then, we concluded that the participants in our study initially engaged in an 
action learning method. Subsequently, the authors engaged in action research through examining 
and reflecting on the extensive body of research literature dealing with assessment of both com- 
petencies and performance outcomes. As a result, it is envisaged that the participants will engage 
in a further action learning stage in 1996. Figure 2 shows the process that was adopted in our 
study. 



STEPS IN MODERATION PROJECT 

Step 1 

Reflection on assessment knowledge and experience 

Collection of examples of tasks, criteria, rubrics and student responses 



Step2 

Shared assessment with colleagues of tasks, criteria, rubrics and student responses provided 
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Step 3 

Incorporation of approved tasks into McAdam collection 

Separation of assessment tasks from teaching materials in McAdam collection 

Step 4 

Shared assessment with colleagues of tasks, criteria and rubrics for modules 111,112 
and 113, Personal Focus, in McAdam’ s collection 

Step 5 

Examination of assessment research and reflection 
Refinement of assessment criteria for tasks, criteria and rubrics 



Figure 2 

The following discussion elaborates on the five steps taken. 

Step 1 

During the first stage of the project the project team prepared and distributed a letter to all 
adult literacy coordinators throughout the State inviting them to participate in the study by 
providing assessment tasks and student responses in the Personal Focus for moderation. The 
response to this invitation was limited. On reflection, one reason may have been that, 
unknown to the project team, Laura McAdam from Logan Institute of TAFE had just previ- 
ously invited TAFE coordinators to provide assessment tasks and lesson materials for the 
Personal and Employment strands with the purpose of collating these in a publication that 
would be available to teachers working in TAFE institutes in Queensland. A second reason 
may be that the timing of the request was too early. While the curriculum, Certificate of 
Vocational Access, was generally introduced during the first semester of 1995, coordinators 
may not have felt confident in sharing their assessment tasks at this stage. 

Step 2 

Despite the paucity of materials that were available for sharing, a workshop was held with 
twenty-five local adult literacy practitioners to evaluate the assessment tasks and student 
responses that had been gathered. A proforma had been developed by the project team and 
was available for use by the workshop participants. The mood of the workshop was very 
supportive of the project. In addition, comments such as: “My standards have been too low 
when assessing students”, and the reverse, were voiced. This session corroborated the con- 
tention by Lyons which was reported in Cope et al. (1994:59) that: 

Moderation helps education and training providers to apply and interpret standards 
within a credential in the same way. It allows us to verify that assessment processes 
and decisions, particularly those critical to the attainment of credentials, are valid and 
consistent - that is, they assess fairly what they are meant to assess and are reliable. 

Step 3 

Following the moderation session, the participants’ responses were examined. Those assess- 
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ment tasks that were rejected were removed from the collection. A further finding was that there 
was no unanimity regarding the value of assessment tasks that were being used by practitioners 
within the TAFE system. Thus, a decision was made to limit the study to assessment tasks, 
assessment criteria and rubrics. 

At this time the project under the direction of Laura McAdam, a teacher at Logan Institute of 
TAFE became known. The aim of this team was to collect assessment items and teaching 
resources relating to the Personal and Employment strands of the Certificate of Vocational 
Access that were being used by literacy teachers in Queensland. A further decision was made to 
incorporate the assessment examples of the Personal strand into our collection. A further task was 
the evaluation and drafting of the assessment criteria sheet that had been used for the moderation 
session in the previous step in the project. 

Step 4 

A second session was held with colleagues to moderate the assessment tasks, assessment criteria 
and rubrics in the Personal Focus aspect of the McAdam collection. Although the proforma that 
had been used in Step 2 was again employed, the task was very time consuming. Those present, 
however, expressed support for the opportunity to participate in this work. 

A further problem which arose was that no specific assessment criteria had been developed to 
ensure that the assessment by the participants in this session could be replicated. This situation 
would appear to be typical of action learning processes. 

Step 5 

Extensive research and reflection relating to the assessment of performance outcomes have taken 
place in the United States, and it seemed crucial that we should access this work with a view to 
adding depth and validity to the project. After examining this literature, the set of assessment cri- 
teria that are listed above were adopted by the project team. 

Future 

It is anticipated that adult literacy coordinators and teachers throughout Queensland will be invit- 
ed to examine the assessment tasks in the Personal Focus aspect of the McAdam collection. The 
results of this further work will be published in a booklet and distributed throughout the State. 

Discussion 

This small research project has added to the body of evidence showing that action 
leaming/research can contribute to both the personal and professional competence of participants. 
Further, by accessing the research literature on the assessment of performance outcomes and 
developing a theoretically-based model for evaluating assessment tasks, criteria and rubrics, the 
work has been sharpened. However, there are wider issues that need to be considered in the 
delivery of adult literacy programs. 

In the introduction, two paradigms of ignorance were noted. This study has dealt with the first: 
nam ely filling in the “blank spots” - taking the student assessment within the Certificate of 
Vocational Access a step further through moderation processes. It is important, as well, to consid- 
er the environment in which this research project is positioned and to examine the “blind spots” 
in current notions of best practice. As noted briefly in the first section of this report, the adoption 
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of a competency-based approach is problematic and the application of the theoretical under- 
pinnings of the Certificate of Vocational Access may not be the most appropriate model for 
the field. 

One of the assumptions on which a competency-based system is based, is that it is possible 
to capture the essential knowledge and skill underpinning performance within pre-specified 
assessment tasks and criteria. At this time, it is not known whether this is possible. The 
continuing work of cognitive scientists, both in the United States and in Australia, is likely 
to inform our practice but there is still much to be learned. A further limitation experienced 
by practitioners in the adult literacy field is our incapacity to distinguish between poor per- 
formances that are due to limited cognitive functioning and poor performances due to limi t, 
ed opportunities (Campione & Brown, 1 990). 

Taking a broader view, the recent emphasis on training for employment appears to be 
unnecessarily limiting in the adult literacy field. Adult literacy programs have traditionally 
catered for the diverse needs of prospective students. These have included family needs 
such as helping children cope with their school demands as well as participating in more 
general community activities. The authors of the National Framework of Adult English 
Language, Literacy and Numeracy Competence (1994) recognised this tradition and built 
into their model the scope for curricula to continue to meet non-employment needs. Other 
researchers such as Porter (1990) and Stevenson (1992) both support the view that focusing 
on employment needs ignores both the personal and social needs of individuals. 

There is recognition, however, in the Australian National Training Authority Act of 1992 
for the promotion of “advanced technical training and further education opportunities” 
(Crombie, 1995:3) for the community as well as for the workforce. These further opportuni- 
ties have been interpreted as continuing throughout a person’s life. How these opportunities 
can best be translated into curricula remains an issue. 

In the case of the Queensland curriculum, the Certificate of Vocational Access, which is 
structurally based on the National Framework of Adult English Language, Literacy and 
Numeracy Competence (1994), the question Auerbach asked at the Australian Council for 
Adult Literacy in Perth in 1 994 is still relevant: Are the four foci of employment, personal, 
vocational and workplace, and the six aspects of identity, group, organisations, community, 
procedures and technology the most useful structures for developing a literacy curriculum? 
While this model was reflective of the field’s understanding at that time, is it still appropri- 
ate? Or might other models serve us better? 

Spady (1994:22-23) has offered one model for consideration. He identifies ten clusters of 
social roles called “Fundamental Life Performance Roles”, that he considers essential for 
successful participation in society. These range from citizen and parent through to employer 
and worker and are consistent with the roles identified in the 1991 SCAN report that has 
been used by a number of states in North America to frame their performance outcome 
strategies. No doubt there are other models that could be considered. 

More thought also needs to be given to moderation issues. In the short term, if acceptable 
levels of parity are to be recognised it is imperative that moderation processes are in place 
to firstly, choose assessment tasks that meet specific outcomes, ensure that their accompa- 
nying criteria and instructions are clear and appropriate, and then determine standards for 
student responses to these tasks. This report has detailed how initial work in this process 




64 



70 



Adult Literacy and Numeracy: Assessing Change 



has taken place. Further steps are now required to ensure that the necessary development work 
occurs. If it becomes obvious in the future that a competency-based approach is too limiting for 
our field and the wider vocational education context, then it will, in part, be in response to the 
action learning and action research work in projects such as this which attempted to fill in the 
blank spots in the assessment of competencies. 

Ann Kelly 
Southbank Institute of TAFE 
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MODERATION OF COMPETENCY-BASED 
ASSESSMENT: ISSUES AND PRACTICE IN 
ADULT LITERACY AND NUMERACY 

John Bailey 
Pam Weston 
Shirley Verrall 
Jeremy Audas 

BACKGROUND 

Competency based training and competency based assessment in the field of Literacy and 
Numeracy have now been incorporated into the curriculum CNL03, Certificate in 
Vocational Access. CBT module learning outcomes usually describe skills, standards and 
conditions of performance. However, this curriculum does not incorporate any explicit stan- 
dards and rarely a performance condition. The learning outcomes are based on the 
ACTRAC National Framework of Adult English Language Literacy and Numeracy 
Competency (1993). The National Framework does not provide a set of Language, Literacy 
and Numeracy standards, nor is it developmental. Therefore, the standards are implied in 
the learning outcomes. This causes great difficulty for teachers designing assessment tasks 
and carrying out assessments. 

In literacy education it is especially important that teachers not become merely transmitters 
of systemic curricula and assessment procedures, but that they themselves play a major role 
in determining learning outcomes and assessment strategies in collaboration with learners 
(Lytle & Wolfe, 1989). A criterion- referenced approach to assessment can meet the 
requirements of accountability (to funding bodies) while also being compatible with a 
learner-centred curriculum (Brindley, 1989). 

The significance of this project is that it provides some applied research into CBT assess- 
ment moderation. Any training provider must be able to demonstrate the consistency of out- 
comes and quality processes to clients. It is essential that Literacy and Numeracy teachers 
be able to demonstrate accountability and educationally defensible processes of moderation 
of assessments within CBT curricula. 

As the curriculum CNL03 is used state wide throughout Queensland this research was con- 
ducted within the framework of TAFE Queensland Assessment principles. 
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INTRODUCTION 
Description of Project 

The project was designed to trial a methodology for moderation of competency based assessment 
in the literacy /numeracy field within the Barrier Reef Institute of TAFE. An action learning team 
was formed to carry out the project. This team was made up of literacy and numeracy practition- 
ers from three campuses of the Barrier Reef Institute of TAFE (Townsville, Burdekin and 
Charters Towers). The three campuses service entirely different regions and clientele. 

The Townsville campus is a large centre catering for a population in excess of 120,000 people. 
Most of those who access this campus are urban dwellers. The Burdekin campus caters for a 
large sugar growing community which means most of the clients who attend classes there are 
from rural backgrounds. The campus at Charters Towers is more isolated and caters for a clien- 
tele from widely scattered grazing and mining communities. 

At these campuses CNL03 is delivered in a number of ways. These include a Volunteer Tutor 
Programme, individual learning centres, and full- and part-time classes. The Personal Focus, 
level 1 A and IB were chosen as these were delivered across the three sites. 

Clients who enrol in CNL03 in Townsville are from English speaking backgrounds as at 
Townsville there is a Migrant Education Unit which caters for the needs of those from Non 
English Speaking backgrounds. In contrast at Charters Towers and Burdekin migrants sometimes 
participate in literacy classes. 

All age groups are represented in the client groups at all three sites. There are also learners with 
physical or intellectual disabilities enrolled in CNL03. This is in keeping with the kinds of stu- 
dents found in TAFE colleges across the states, however, because of numbers in the smaller cam- 
puses the classes can often be of mixed ability. In all of these classes there were students enrolled 
in levels 1 A and IB which further supported the research team’s choice of these modules for 
moderation. 

After an initial meeting, members of the research team returned to their respective campuses to 
design assessment tasks. These were trialed and then at subsequent meetings they were analysed 
and moderated before arriving at a final draft of the assessment items which were compiled into 
booklet form for further trials. The outcomes of these trials led to the research team undertaking a 
detailed process of moderation before producing the final booklet of assessment items. 



PROCESS 

An initial meeting was held to discuss the project. Operational procedures were agreed on and it 
was decided that each participant would keep a reflective journal. Timelines for the project were 
discussed and agreed upon. Minute taking was shared and an electronic white board used to keep 
a record of proceedings. It was felt that we could modify the process according to changing cir- 
cumstances, this being an essential component of action research. Release time for teaching staff 
was negotiated and the issue of transport for members of the group from outlying campuses was 
resolved. Six teachers from across the Institute discussed the methodology of action research and 
a National Staff Development video on action learning was viewed. Issues in the video were dis- 
cussed and proved to be very helpful in understanding the kinds of processes which are appropri- 
ate to an action learning project. 
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At this point we decided on the modules from CNL03 for moderation and the development 
of assessment tasks. As the majority of students were enrolling in the lower level modules 
we would focus on LIT1 1 1 and LIT1 12, level 1 A and IB from the Certificate in Vocational 
Access, Personal Focus. 

The group then discussed the TAFE assessment principles document and examined the 
English Language Training Programs Model Language Assessment Tasks. This revealed 
that these would provide a good model for developing literacy assessment tasks but would 
need modification in terms of the actual tasks to be included in literacy assessment. 

The next activity was to analyse the criteria for developing assessment tasks. The main 
issues here were the need to use authentic texts, acknowledgment of sources and copyright, 
meeting the TAFE assessment principles, ensuring that the tasks were valid and reliable and 
making explicit the standards and conditions for each task. 

Much discussion and negotiation took place at this stage of the project and, in fact, through- 
out the life of the project. There was constant reflection and review of what we were doing 
and where we were going. This was seen as an integral part of the process and added to the 
sharing of information and skills. 

It was emphasised that teachers must be assisted in developing good management strategies 
when implementing assessment tasks with students. These need to be in line with Quality 
Assurance policies and procedures at respective sites. 

The second meeting focussed on an examination of the various assessment tasks related to 
LIT1 1 1 and LIT1 12 (level 1A and IB, Personal Focus) which teachers had brought with 
them. Some of these tasks were developed by local teachers and others were from other 
curricula. One member had already designed a holistic task which assessed all six outcomes 
of LIT1 1 1 in one instrument. The ensuing discussion clarified some of the issues at stake, a 
great number of positive comments about the holistic model added to the impetus to design 
tasks which were “student friendly”. It was agreed that as far as possible, without being 
contrived, the tasks should not address individual outcomes but deal with more than one 
outcome at a time. This was in keeping with the TAFE assessment principles and teachers 
thought that from the student’s point of view it was desirable to use this approach. It was 
refreshing to hear teachers talking about assessment being student centred and not just a 
way of meeting the teachers’ (and the curriculum’s) administrative needs. 

There were some animated discussions about how best to present the assessment tasks in 
their published form, that is, how the final product layout should look. Should we include 
the entire wording of the outcomes as it appeared in the syllabus document or could we 
modify this to suit our needs? It was decided that we really had no authority to modify the 
wording. These discussions were kept at a forceful yet friendly level which seemed to indi- 
cate the degree to which assessment was a contentious issue in literacy circles. 

From time to time the group, in its enthusiasm during these discussions, would get offtrack 
but someone would always bring this to the attention of the others and we would refocus. 
These moments allowed us to become aware of the wide range of issues associated with the 
development of assessment tasks and how the views of all participants were valued as con- 
tributions to the whole project. In addition it indicated that the action research process was 
working as it should, being reflective, self monitoring and using all the skills and abilities 
of the participants. 
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The following five meetings were clearly focussed on the task and although there were many side 
issues of interest to the whole group we were able to maintain focus and complete the require- 
ments for each meeting. All of the meetings generated a better understanding of literacy assess- 
ment tasks as well as reinforcing the cyclical nature of action research. In between meetings tasks 
that were developed in the second and subsequent meetings were trialed with currently enrolled 
students at the various campuses. This enabled the group to analyse each others’ experiences with 
the tasks given to their students which further clarified not only the shortcomings and benefits of 
the various tasks but also the group members’ perceptions of assessment and its role in literacy 
learning. Once the assessment items had been modified and trialed the final format of their publi- 
cation was decided upon. It had to be clearly emphasised that the product we were developing 
was a model of literacy assessment tasks and would need to be adapted for teachers to use in 
other locations. For example, one task required students to read a bus timetable - for the sake of 
authenticity a timetable appropriate to the local environment would have to be substituted for the 
one suggested in the model task. 

When the final meeting had been held we all agreed that the tasks we had developed and trialed 
were appropriate and that the students who had attempted them had found that they were com- 
mensurate with their abilities. They were certainly deemed to be valid in that they measured what 
they were supposed to and reliable to the extent that the variety of students undertook the tasks 
and were able to complete them at similar levels. 

The next step was to employ a desktop publisher to produce the tasks in a booklet form which 
was easy to use with clear guidelines and instructions. 

OUTCOMES 

From entries in the journals kept by the participants during the course of the project it is obvious 
that there is a real need to customise assessment tasks to suit local conditions. With regards to the 
holistic task concerning the snack bar all centres noted that students were able to complete the 
tasks successfully as ordering food from a snack bar is something they are familiar with and they 
were able to relate the activity to an actual place they knew. 

However, with regards to tasks that required knowledge of timetables and suburbs, students in 
the smaller centres had problems completing the tasks as they were not familiar with the concepts 
involved. At one centre, in order to complete the exercise, the teacher had to explain what a sub- 
urb was and how boundaries were formed. This led to other exercises such as map reading. The 
teacher also found it necessary to teach her students how to read timetables. Because these kinds 
of activities are not of everyday concern to her students the teacher involved thought that perhaps 
what had been taught would not be retained for long. This reinforces the idea that literacy assess- 
ment tasks require adaptation to the local environment in order to increase their relevancy. 

It was noted by teachers in each of the centres that students responded best to tasks that were of 
personal relevance to them. This raises the question of how to design and moderate assessment 
tasks that can be used in different places for different kinds of clientele. 

During the course of discussing the implementation of the tasks contained in the booklet it soon 
became clear that teachers had problems with constructing and delivering assessment tasks based 
on given learning outcomes, particularly those of the Personal Focus strand. It was noted by all 
involved in the project that the majority of students we deal with enrol in literacy courses and the 
Individual Learning Centres for their own purposes. They are adults and in the main know what 




71 



Language Australia 



it is they want to learn and they have a wealth of hitherto unrecognised previous lea rnin g. 
For these reasons it is important that assessment tasks need to be related to the individual 
needs of the students. Tailoring these needs to suit given learning outcomes is not an easy 
task. 

It was decided that the best way to achieve this was to discuss the stated outcomes in rela- 
tionship to the students’ own particular needs and the training content. This takes time and 
how tasks could be moderated across an Institute such as the Barrier Reef Institute of 
TAFE, let alone the TAFE Queensland system was of concern to the teachers involved in 
the project. 

From these discussions some important questions arose. One concerns the desirability of 
designing generic tasks for literacy students. It was thought that it would be more appropri- 
ate for assessment to become an integral part of the teaching process and that both the con- 
tent of the course and the assessment tasks be negotiated with the students before the course 
begins. Another concern expressed was that some teachers felt outcomes were not able to 
be taught to enable students to complete the assessment tasks. They felt uncomfortable with 
the idea that teachers could observe students’ day to day work and tick off successfully 
completed criteria rather than have students undertake a more formal assessment. 

ISSUES 

One of the main issues raised by members of the research team was that of Competency 
Based Training and its applicability to adult literacy teaching. While it is possible to match 
the needs of individual students to the generic outcomes offered in CNL03, The Certificate 
in Vocational Access, it is often not the most efficient or effective way of addressing stu- 
dents’ needs. The ability of literacy teachers to develop a range of reliable and valid assess- 
ment instruments to match the outcomes of the course and remain relevant and in an appro- 
priate context requires a huge amount of work, even if models of assessment are available. 

It could be argued, quite rightly, that this is part and parcel of teaching yet it does demand 
an extraordinary amount of work when compared with developing assessment instruments 
for say, an office studies course. No single instrument will be relevant for every student in 
every context. That is why the group has emphasised the idea that the assessment items 
developed for this project are models only and require reworking and adapting to suit the 
needs of a wide range of students with differing needs. 

Perhaps Competency Based Training is not the best way of teaching and learning literacy 
and a more student centred and contextually appropriate method might be devised, particu- 
larly when no industry standards for literacy exist. The curriculum currently being used in 
Queensland was developed from the National Framework of Adult English Language, 
Literacy and Numeracy Competence (ACTRAC, 1993) and was certainly developed along 
sound theoretical lines but in putting theory into practice the curriculum has become some- 
thing of a behemoth - problematic in terms of the way outcomes are stated and difficult to 
implement in terms of assessment processes. 

The research group wrestled with these ideas and much discussion was generated. It was 
clear on reflection that the action learning process enabled the group to express widely held 
views and beliefs in a supportive environment enabling issues such as the above to be 
expressed to colleagues. We realised that we were all in the same boat with regards to 
assessment of literacy students and that it is very easy to steer the course of least resistance 
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which may not be in the best interests of students. As has been mentioned previously in this 
chapter, literacy students come to classes with a wealth of prior learning with no two students 
being alike in the context of that learning. To enable teachers to assess appropriately it was felt 
that the best we could do was to encourage teachers to use adapted models of assessment, howev- 
er, even with the best of intentions the adaptations of existing models might still not meet the 
needs of individual students especially if prior learning is not taken into account. 

Another issue of note, and not concerning the task but the process, was the amount of time 
required for action research. We felt that the time we had was limited and that there was pressure 
to deal with the immediate task. More time for general discussion would have been appreciated 
because this kind of discussion often uncovers some of the deeper issues pertaining to the task, 
especially philosophical issues related to literacy learning and assessment. Certainly discussion 
continued outside the meeting times but this was limited due to pressure of work and geographic 
encumbrances. Perhaps our first meeting could have been devoted to a discussion of our own 
philosophical positions. 



John Bailey 

Townsville Campus - The Barrier Institute of TAFE 

Pam Weston 
Charters Towers Campus 

Shirley Verrall 
Burdekin Campus 

Jeremy Audas 
Townsville Campus 
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Appendix 



THE ASSESSMENT EXEMPLARS 



INTRODUCTION 

Competency based training and competency based assessment in the field of literacy and 
numeracy has been delivered in TAFE Queensland since June 1994. The competencies are 
designated as Learning Outcomes which are based on the ACTRAC National Framework of 
AduLt English Language Literacy and Numeracy Competency (1993). The National 
Framework does not provide a set of Language, Literacy and Numeracy standards, nor is it 
developmental. Therefore the standards are implied in the learning outcomes. This causes 
great difficulty for teachers designing assessment tasks and carry out assessments. 

This booklet was produced as the result of a small project trialing an action learning 
methodology. A team of practitioners of language and literacy modules representing three 
sites of the Barrier Reef Institute of TAFE (Townsville, Burdekin, Charters Towers) 
designed and trialed assessment tasks in at least one class at each campus. These “classes” 
included face to face and independent learning centres. 

The team met on three occasions. Firstly, to plan the project. Secondly, to devise assess- 
ment tasks to be trialed across the Institute. Finally, to modify the original tasks based on 
student responses to them. The tasks are related to Modules LIT1 1 1 and LIT1 12, Literacy 
and Language, Personal Focus - levels 1A and IB as these were the only modules offered at 
all three sites. 

Those involved in the project were: 

John Bailey - Facilitator (Townsville) 

Pam Weston (Charters Towers) 

Shirley Verrall (Burdekin) 

Jeremy Audas (Townsville) 
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CNL03 MODULE PLANNER CHART 



1 , 
Module 


LIT111 LITERACY AND LANGUAGE PERSONAL FOCUS • LEVEL 1A 


Learning 
I Outcome: 1 


Read and write very short, explicit, personal information related to the learner’s identity and relevant to individual goals, e.g. 
personal bills, appointment cards, personal details card, date of birth, a greeting on a card, expressing opinion. 


Assessment 


a) 


Complete a clearly set-out form with family name, first name, address, age, or next of kin, according to the following. 


Criteria: 




=> 


write legibly in block letters 






=> 


write the details accurately from memory 






=> 


spell with a reasonable degree of accuracy 




b) 


Interpret an account: 






=> 


slate the amount owing 






=> 


slate the date of the account 






=> 


identify who the account is from 






=> 


show an understanding of what the account means, 
e.g. I owe the doctor S22.00. 


Assessment 




=> 


written task la. 


Method; 




=> 


short oral answer lb. 


Assessment 




The learner will be provided with: 


Conditions: 






a teacher/leamer selected, relevant document (realistic or simulated) requiring completion of basic personal details, e.g. 
personal details form for class records. 








a teacher/lcamcr selected familiar account which is explicit and clearly formatted, e.g. doctor s bill, tradesperson s account. 






The learner will provide: 






=> 
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Module 


LIT1 1 1 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL 1 A 


Learning 
Outcome: 2 


Read and record very short, explicit, context-embedded texts associated with membership of and interaction within a relevant group 
e.g. in the family, community group; members of class; friends. 


Assessment 

Criteria: 


a) Prepare a list of names (minimum 3) of significant members of relevant group in order to complete a relevant task, c.g. guest list for 

party, Christinas card list; list of names of class members for tea and coffee; list of family members for Christmas shopping 
according to the following: 




=> write the names from memory 




=> write the names so that they can be understood 




=> look up phone numbers and record accurately 




=> explain how the spelling of names will be checked before writing. 


Assessment 


=> practical task - look for names in phone book. 


Method: 


=> written task - record names 


Assessment 


The learner will be provided with: 


Conditions: 


=> a teacher identified situation relevant to the learner. 




The 1 earner will provide: 








■St 
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CNL03 MODULE PLANNER CHART 



| Module 


UT111 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL 1 A 


J Learning 
I Outcome: 3 


Explain the meaning ofveiy short, explicit, pictorial texts which are relevant to the learner's interaction with organisations within 
ihc community, their goals structures and activities c.g, safety signs, fire exits, bus routes, maps. 


j Assessment 
| Criteria: 


a) Interpret three relevant texts with pictorial/visual clues e.g. map of the building, business hours open, fire oft, warning signs small 

shopping centre map according to the following: 6 6 * 




=> predict meaning of some unknown words aided by the context, the format and symbols 
=> identify correctly some key words 




=> provide an adequate explanation for safe usage, as appropriate e.g. which f.re exit (o use; which assembly point to use how to 
find a shop in shopping centre. ’ 




=> explain how to use the text e.g. bus timetable, local shopping centre map. 


Assessment 

Method: 


=> practical task 
=> oral explanation 
=> simulation 

=> role play 


Assessment 

Conditions: 


The learner will be provided with: 

=> three leachcr/leamer selected texts which are relevant to the student’s goal. 

The learner will provide: 

=> nil. 
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Module 


LITlll LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL 1A 


Learning 
Outcome: 4 


Perform unproblematic, routine oral interactions, such as giving or receiving concrete pieces of information, with relevant members 
ofthe community e.g, telephone for an appointment, request information re opening hours. 


Assessment 

Criteria: 


a) Perform a simulated oral interaction to gather a single piece of information, date and lime, request appointment | 

=> explain clearly the information to be given or required so that the listener understand and respond appropriately. 

=> report orally the outcome of the transaction e.g. the appointment is with Dr. Smith at 2.00pm on Friday 6 May. 

=> interpret the recorded information (e.g. amounts, name, time) accurately. 

=> describe the strategies used to read materials, to interact with the listener and to record the information: c.g, planning, 

quesuons to ask, use of format to find a logo, name and address, matching, rehearsal, greeting, confirmation, repetition, 5W 
strategy (who, when, where, what, why). 

=> evaluate the performance. 

=> use appropriate rules of social etiquette re opening and closing transaction. 

b) Report and evaluate orally an actual oral interaction that the learner has carried out 


Assessment 

Method: 


=> oral explanation 

=> oral interaction 

=> practical task 
=> written task 
=> role play 
=> stimulation 


Assessment 

Conditions: 


The learner will be provided with: 

=> a teacher/ learner identified situation relevant to tire learner’s interaction with the community; 

=> an oral description of a scenario relevant to the learner’s current role within the community e.g. the need to make an 

appointment (with doctor, for a driving less, etc); the need to check opening hours of local library the name nnd phone number 
of the contact or single-step instructions on how to find the phone number, given the written contact name; 

=> how to find the phone number given the written contact name; 

=> pen and paper to record information; 

=> a telephone to make an actual or simulated dialling; 

=> a partner who will be co-operative and provide the requested information clearly and will repeat as required (e.g. spell surname, 
street, name suburb). 

The learner will provide: 

=> nil. 
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CNL03 MODULE PLANNER CHART 



Module 


LIT111 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL 1 A I 


Learning 
Outcome: 5 


Use very short, explicit, pictorial/symbolic language, literacy and numeracy texts related to commonly used procedures for | 

completing relevant tasks to meet the learner’s individual goals, c.g. short recipes, safety procedures, voting cards, driving guide. | 


Assessment 

Criteria: 


a) Follow written directions involving pictorial or symbolic language, to carry out 2 relevant tasks, one requiring a written completion 

task, e.g. follow an uncomplicated recipe, and complete procedures for cleaning floors. 




=> show adequate understanding of tire text to be able to describe the procedure correctly to the teacher or other student 




=> use strategies such as sequencing and copying to complete single word entries 


Assessment 


=> simulation 1 


Method: 


=> written task I 




=> oral explanation | 


Assessment 


The learner will be provided with: | 


Conditions: 


=> two procedural texts. I 




The learner will provide: I 
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Module 


LIT111 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL 1A 


Learning 
Outcome: 6 


Use very short, context-embedded pieces of text related to using new technology and aided by instructions, diagrams and observation 
of model demonstration according to the following: 


Assessment 

Criteria: 


a) Describe or perform a task using selected technology and aided by instructions, diagrams and observation of model demonstration 

according to the following: 

=> read or predict the meaning of relevant words, 
c.g. on, operate, danger, avoid contact with water 

=> explain the process and the necessary safety procedures orally 


Assessment 

Method: 


=> oral explanation 
=> simulation 


Assessment 

Conditions: 


The learner will be provided with: 

=> an item of new technology, c.g. equipment such as microwave, machines, photocopier, calculator, video machine, 

fax machine, automatic food machine, requiring no more than 3 operations and having single word, written warnings, 
diagrams or instructions; 

=> an initial demonstration of the use of selected tcclmology. 

The learner will provide: 
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CNL03 MODULE PLANNER CHART 


I Module 


LIT112 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL IB 


I Learning 
| Outcome: 1 


Use short pieces of oral and written text related tojdentify and personal needs, e.g. expressing own idcas/opinions (spoken and 
wntten), personal details form, diary, sharing family history, setting goals, routine resume. 


| Assessment 
j Criteria: 


a) Complete a set-out form with personal details, past employment, and schooling according to the following: 

=> wnte the details accurately from memory 
=> spell with a fair degree of accuracy 

=> use resources to find out conect spelling, e.g. telephone book, personal records. 


Assessment 

Method: 


=> written task la. 


Assessment 

Conditions: 


The learner will be provided with: 

=> a relevant form requiring completion of persons! details, past employment and schooling. 

The learner will provide: 

=> nil. 
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Module 


LITI12 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL IB 


Learning 
Outcome: 2 


Participate in a relevant group interaction using oral and written texts that arc short and highly structured e.g. family community 
group, friends, colleagues, discussions, list of members, diary entry about a meeting. 


Assessment 

Criteria: 


a) Read written names relating to membership of a group, e.g. family names, work/team members. 

b) Observe social conventions of group oral interaction 

c) Recount orally to a group, the details of an incident or previous experience, e.g. yam, recount of interesting incident, accident, report 
according to the following: 

=> provide sufficient background and detail for the listeners to understand (Student to recount incidenl/accidenl/cvcnt using cue 
cards for names of people involved). 

=> use appropriate vocabulary and language 

=> provide eye contact with group members and use appropriate gestures 
=> communicate clearly using appropriate volume and enunciation. 


Assessment 

Method: 


=> practical task 

=> oral explanation 


Assessment 

Conditions: 


The learner will be provided with: 

=> an actual supportive group of people who arc relevant to the learner's goals and needs e.g. class group. 




The learner will provide: 
=> nil. 
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CNL03 MODULE PLANNER CHART 



Module 


LITin LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL IB 


Learning 
Outcome: 3 


Use documents that have a defined structure and relate to the learner’s interaction with organisations, c.g. rates notice, bills, 
permission notes from school, driver’s licence renewal form, bank forms, timetables, to carry out routine tasks of gathering 
information and observing relationships, with the aid of format, symbols, numbers and known vocabulary. 


Assessment 


a) Read two documents with defined structure, c.g. car registration, cheque book, school excursion notice 


Criteria: 


b) Respond in writing to same two documents, c.g. cheque, note according to the following: 




=> explain orally the purpose of the document and locate the main pieces of information 
=> respond in writing to the document, c.g. writing a cheque 

=> record written information accurately, using proforma or appropriate layout of written communication 
=> provide correct information and spe lling. 


Assessment 

Method: 


=> practical task 
written task 
=> oral explanation 
=> simulated situation. 


Assessment 


The learner will be provided with: 


Conditions: 


=> one relevant document, c.g. registration notice form, one cheque proforma. 




The learner will provide: 
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Module 


LIT112 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL IB 


Learning 
Outcome: 4 


Use short, highly contextualised relevant materials from the community to complete a range of language, literacy and numeracy tasks 
relevant to the learner’s goals, c.g. catalogues, brochures, business cards, newspaper, advertisements, invoices, uncomplicated notes 
to and from school, short telephone messages, following oral directions from chemist, club membership forms. 


Assessment 

Criteria: 


a) Make a simulated telephone call to a service c.g. plumber according to the following: 

=> follow social conventions of telephone calls such as greetings and courtesy 
=> record information accurately and legibly; times, dates and cost 
=> check numbers and details as appropriate. 


Assessment 

Method: 


=> simulation 
=> written task 


Assessment 

Conditions: 


The learner will be provided with: 

=> a telephone message proforma, provide pen and note paper 
=> a teacher or tutor to play role of plumber. 

The learner will provide: 

=> nil. 
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CNL03 MODULE PLANNER CHART 



Module 


LIT112 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL IB 


Learning 
Outcome: 5 


Use short, explicit, literacy and language texts related to known procedures for completion of routine tasks relevant to the learner’s 
goals, e.g. routine recipes, instructions on medical products, familiar procedures. 


Assessment 

Criteria: 


a) Write a procedure of 5 or more steps, for a frequently performed task, e.g. how to get from A to B, how to change a tyre according to 

the following: 




=> 


provide sufficient information clearly 




=> 


order the steps correctly 




=> 


use an appropriate format for a procedural text, e.g. action statement 




=> 


write steps to perform task with 5 or more steps, e.g. cliangc tyre, make tea, do washing and make checklist for teacher from 
these. 




=> 


include safety factors where necessary 




=> 


approximate spelling and use strategies to correct spelling so that the text is understandable. 


Assessment 

Method: 




written task 


Assessment 


The learner will be provided with: 


Conditions: 








The learner will provide: 


L 




a relevant procedural task. 
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Module 


LIT112 LITERACY AND LANGUAGE PERSONAL FOCUS - LEVEL IB 


Learning 
Outcome: 6 


Use short, relevant, explicit, clearly formatted, literacy and language texts related to technology, e.g, microwave, library index, 
automatic teller machine, watch, camera, photocopies. 


Assessment 

Criteria: 


a) Instruct a tcacherAutor on the use of a machine, e.g. use of fire extinguisher (without operating); use of fax machine, electric stapler, 

microwave, fast food machine, calculator according to the following: 




=> show evidence of logical sequence and safety awareness 




=> speak dearly 




=> develop checklist for teacher to complete as part of assessment 




=> explain the meaning of necessary words and symbols on the apparatus, e.g. safety/warning signs, sequence of operations. 


Assessment 


=> practical task 


Method: 


=> oral explanation 


Assessment 


The learner will be provided with: 


Conditions: 


=> a relevant technology, if necessary 




The learner will provide: 




=> a relevant technology. 
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Assessment Items for CNL03 Module LIT111 Level 1A 

The assessment item for this module is holistic and addresses all six outcomes for the module. Individual items 
may need to be modified to suit the needs of the local environment and to ensure that they are in an authentic 
context for students. 



MODULE: LIT111 - LEVEL 1A LO 1,2, 3, 4, 5,6 

You want to apply for a part-time job at Sam's Snackbar: Fill in the following sections of the form below 



Assessment 

Criteria 



LO la) 



LO 2 a) 



APPLICATION FOR EMPLOYMENT E2.2 
(USE BLOCK LETTERS) 



Sam’s Snackbar 

PO Box 452 

MOUNTAIN VIEW Q 4821 

I wish to apply for employment at Sam’s Snackbar in 



(suburb) 

DATE OF APPLICATION _ 



(town/city) 



(day) 



(month) 



PERSONAL DETAILS 
SURNAME 



GIVEN NAMES 



Head Office 
PO Box 151 

BRISBANE QLD 4000 
Telephone: (07) 369-7156 
Fax; (07) 369-7157 



(year) 



PHONE 


DATE OF BIRTH- 




(day) 


(month) 


(year) 


NEXT OF KIN 


NAME 






INFORMATION 
(for emergency purposes) 

PHONE 


ADDRESS 






RELATIONSHIP 






REFEREES 








n NAME 






PHONE 


2) NAME 






PHONE 


3) NAME 






PHONE 
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Assessment 

Criteria 



LO 4 a) 



L04b) 



L03 a) 



MODULE: L1T111 

Congratulations! You have received a letter from the Manager of Sam’s Snackbar 
asking you to phone in for an appointment 



Ask your teacher to arrange for you to do that on the internal phone. 

TIME OF APPOINTMENT 

(dak) (time) 

Tel i one of your classmates what time your appointment is. 

Consult the bus timetable to decide which bus you should take to be there in time. 
Tel 1 your teacher how you found the information and why you decided on that time. 



Use the bus timetable to explain to your teacher which bus you will need to catch 
to get to your appointment on time. 

MOUNTAIN VIEW BUSLINE 

MONDAY TO FRIDAY 
Acacia Ridge - Mountain bTeiv 

DEP ARR 

6.30am 7.00am 

7.00ara 7.30am 

7.30am 8.00am 

Thereafter buses depart on the hour until 4.30pm. 

4.30pm 5.00pm 

5.00pm 5.30pm 

5.30pm 6.30pm 

Thereafter buses depart on the hour until 10.00pm. 
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Assessment 

Tasks 

L03a) 



L03 a) 



MODULE: LIT 111 



Terrific! You got the job. Here is a map of the shopping centre and Sam’s 
Snackbar is in the Eatery. 

Referring to the key, show your teacher around the shopping centre. 




1 Sarah’s Beauty Salon 

2 Style Ladies Fashions 

3 Sports Goods 

4 Computer Spot 

5 Sparky’s Electrics 

6 Sam’s Snack Bar 

7 Fur, Feathers and Fins 



One of the things to do 
is to find out when the 
Snackbar is open. 

Read the sign opposite 
and tell your teacher 
when the Snackbar 
is open. 



8 Supermarket 

9 Newsagents 

10 Eatery 

1 1 Toilets and Parents’ Room 

12 Centre Management 

13 Cleaners 

14 CarPark 



Sam’s 
Snack Bar 

OPEN 
8 am - 6pm 
Mon to Fri 

8am - 2pm 
Saturday 

CLOSED 

Sunday 
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Assessment 

Tasks 

LOlb) 



L06 



MODULE: LIT 111 



You have had some electrical work done. What kind of a form is this? Explain it to your 
teacher. 



SPARKY’S ELECTRICS 

Electrical Good 
Electrical Contractors 

PO Box 378 MOUNTAIN VIEW Q 4821 
Telephone: 872 1143 



Invoice No. 5844 



to 



Please note: 

Our terms are now 
STRICTLY 14 DAYS 
from date of invoice 



Select one of the machines or appliances listed below and demonstrate how to use it. 

As you use it, explain two safety features you have read in the user’s manual. 

Explain how you make sense of difficult words or instructions in the user’s manual. 

The teacher will select three phrases or words from the manual. Explain what they mean. 



Sandwich Maker 
Calculator 
Microwave 
Photo Copier etc. 



O 
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Assessment 

Criteria 



L05a) 

L05a) 



MODULE: LIT 111 

YELLOW BUTTERCAKE 

CAKE MIX 



1 PRE-HEAT: gas or electric oven 180° (350°F) or 160°C for a fan forced oven. 
Grease and flour cake pan or lightly grease non-stick pan. 



2 PLACE 
Cake Mix 

1 egg 
2/3 c water 

3 tabsp soft 
butter or 
margarine into 
a mixing bowl. 



3 MIX 

with a wooden 
spoon until 
ingredients are 
combined 
(approx 30 
seconds). 



4 BAKE 

One 22cm x 1 1cm loaf 
pan 40-45 minutes .... 

OR One 18 cm sandwich 

pan 35-40 minutes 

OR shallow patty pans 
[(Bake at 190° C (375°F)] 
12-15 minutes. 

Cake is baked if it springs 
back when lightly 
touched in the centre. 
Allow cake to cool in pan 
5-8 minutes before 
inverting onto a cake 
cooler. 



Tell your teacher, in ymir own words, how you would mak6 this cake. 
Procedures for cleaning the floors. Arrange in the correct order. 

Mop the whole floor area. 

As each section of the floor dries, replace all furniture, 
bins and boxes to their correct places. 

Clear all moveable items from the floor (chairs, bins, boxes etc) 
Fill a bucket with hot water and add one capful of Super Cleano. 
Vacuum the floor one section at a time. 
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Assessment Items for CNL03 Module LIT112 Level IB 



The assesment items for this module address individual outcomes. Learning outcomes 2, 3, 5, and 6 require 
negotiation with students and use authentic texts and technologies to assess competence. 

L02- is an oral assessment and should be developed by the teacher in consultation with the student(s). See 
module planner chart for details. 

L03- requires the student to complete a written task using authentic texts with which the students are familiar. 

LOS- requires the student to write a simple procedure and should be negotiated with individual students see 
module planner. 

L06- requires the student to demonstrate, with oral description, how to use a piece of technology. See module 
planner for detail. 



O 
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Emphasis should be on developing appropriate tasks in context which may vary from student to student. 



LO No 1 LIT112 



APPLICATION FOR EMPLOYMENT 

(PLEASE PRINT OR TYPE. DO NOT WRITE) 

1. Position Desired 





Mr 


□ 


(mark with 


2. Full Name (BLOCK LETTERS) SURNAME 


Mrs 


□ 


an X 




Miss 


□ 


whichever 




Ms 


□ 


applicable) 



(Christian or Given Names) 
3. Address 



(Maiden Name or Change of Name) 



Telephone No. Home_ 
4. Date of Birth 



Business 



Place of Birth 



5. Citizenship: Australian Citizen 

6. Marital Status: 

7. Next of Kin: 



Other (please specify) 



Address: 



8. EDUCATION HISTORY 



(a) 



SECONDARY EDUCATION: Highest level achieved or being attempted: 



Name of Examination e.g. Year 10, 11, 12 (Matric) 



Period of Attendance 19 to 19 



Study Record - 
List all subjects 
Attempted & Results 

Name of School: 


Subject 


Result 


Subject 


Result 





























(b) 

POST SECONDARY Course of Study: 



List all 
Subjects 
Attempted 
& Results 


Subject 


Result 


Subject 


Result 


Subject 
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9 . EMPLOYMENT HISTORY(lncluding Part Time and Vacation Employment) 



Period of Employment 
From To 



Position 



Name of Employer 



Nature of Work 



Reason for Leaving 



10. HEALTH Are you normally in good heatth? YES □ NO □ (Mark with an X where applicable) 
Heave you any physical disabilities/ If Yes, give details: ; 



11. OTHER QUALIFICATIONS (List any further information related to your application including further study, sporting clubs, social 

or community interest and membership of clubs or societies, etc): 



12. MILITARY SERVICES: Are you a former member of the Australian Forces? YES □ NO □ 

13. OTHER INFORMATION (List any further information related to your application including further study, sporting clubs, social 

or community interests and membership of clubs or societies, etc): 



I declare that the above information is true and correct 
DATE / / 



(Signature of Applicant) 
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NAME DATE 

TASK Make a telephone call to arrange a service. 



Telephone the plumber to come and fix your blocked sink. 
Arrange a time and day. The plumbing company is called 
Inland Plumbing. They charge $55 per hour plus materials 
You should ask their hourly rates, and remember to give 
your address. 



Plan your phone call with a partners help: 



YOU 



INLAND PLUMBING 



Ring the number 



Give name 



State purpose 



Record the service details 
(e.g. times, dates & costs) 



Repeat to confirm 



Close 
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CLASS ASSESSMENT SHEET DATE 

TASK Make a telephone call to arrange a (plumbing) service. 

( /or x) 


Confirmed details 
appropriately 


























Intelligible 

pronunciation! intonation 
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Good 

vocabulary 

choices 


























Turn taking in 
oral interaction 


























Stated 

purpose 

clearly 


























Opened & 
closed politely 
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LET’S START READING: THE FIELD 
TRIAL OF A COMPUTER PROGRAM AS 
AN AID IN ADULT LITERACY TEACHING 



Maureen Mortimer 



This chapter reports a study of reactions of students to a multimedia package I have been 
designing. It is also a record of individual progress and achievement as they used the pack- 
age as part of their adult literacy tuition. 

1. Reasons for the study 

1.1 Client group 

Whitsunday is a small college comprising two campuses, Bowen and Cannonvale, 65 kms 
apart. Both campuses have a similar problem, that of a largely itinerant population. Many of 
these people would prefer a more permanent lifestyle but lack the education and skills to 
hold down the type of job which would make this possible. The employment they manage 
to pickup is casual and mostly on a ‘drop everything and be here within half an hour if you 
want the work’ basis. This means that classes are often missed. The groups also have a 
wide variety of ability which makes teaching difficult. This is not unusual 

It is already estimated that between 10 and 20 percent of our adult population in 
Australia require extra help to reach a state of functional Literacy. The spectrum of 
needs can range from being totally unable to read and write to those whose verbal 
skills are so eroded as to make communication very difficult. (Taylor & Thommassen 
1991) 

A need was seen for more individual activities to back up classroom work. This would 
mean that students would be able to still work on their own individual literacy programs 
and not be restricted to attending classes with a rigid timetable. 

Two matters concerned me about the adult literacy students, one was a lack of motivation 
in some students often manifested in a lack of wanting to participate in class activities, and 
a related phenomenon, that of irregular attendance. These students lack selfesteem and do 
not mix with other students in the college. I was interested to see if these behaviours could 
be changed. 
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1.2 Why use a Computer? 

Interviews with the students and research into the potential job market revealed that it was not 
only lack of literacy skills which was a handicap to getting that elusive job, potential employers 
were looking favourably on those who had computer knowledge, however minimal that knowl- 
edge might be. 

The concept of useable knowledge is constantly changing and being replaced by other ‘new 
technologies’ and literacies associated with micro electronic technology. 

To get jobs in the new information based industries people must be able to interpret and 
manipulate the new symbols and icons that are displayed in the new technologies. (Kell 
1995) 

Computers are also widely in use in other courses being offered in the college. By extending the 
use of computers in adult literacy classes we were helping our students identity more with the 
other students in the college rather than thinking that they were the ‘poor relations’ or ‘the dum- 
mies’ as many have referred to themselves. It has been found that the use of computers has many 
advantages, including: student control over their own learning, they have active involvement and 
can vary the pace and content of their learning. The regular immediate feedback given by the 
computer is appreciated by students as is the constant record kept of their improvement and the 
sense of privacy as no-one else could see their mistakes. One important facet of the introduction 
of the computer with adult literacy students is the improvement in their self esteem as they over- 
come their fear of technology and really feel that they are learning a new marketable skill. 

2. Designing the program 

2.1 Points to consider in the design of the program 

ARIS recently published a reproduced a set of guidelines on evaluating literacy and numeracy 
educational software. 

These guidelines were developed by a committee with representation from TAFE NSW 
(Foundation Studies Tr ainin g Division, NSW Adult Migrant English Service, the Board of 
Adult and Community Education and the NSW State and public libraries. The committee 
was convened by the State Library of NSW) (ARIS, 1995) 

The following is a summary and my interpretation of these guidelines in the context of my pro- 
gram. 



2.1.1 Pedagogy 

The program should: be clear in its objectives and follow the scaffolding and fading tech- 
nique of strong support gradually leading to independent study. The student should be 
aware of the learning outcomes of the program which progresses from known simple mater- 
ial to unknown challenging tasks. 

2.1.2 Content 

This should be free of all stereotypes, it should also be fun to use. Appropriate audio should 
back up the visual components of the program. I wanted to avoid synthesised sound and to 
use real warm friendly voices. 
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2.13 View of Language 

The words to be learned should be presented in context, preferably in a context which 
is meaningful to the students and appropriate to their daily lives. 

2.1.4 View of Learners 

The students are all mature adults, the program and material should not treat them as 
children. Many are computer illiterate but this should be overcome by instruction in a 
manner fitting to adults. Games are a good teaching medium but I was anxious that 
our students did not feel that they were being patronised. 

2.1.5 Feedback 

This should be immediate and whenever possible positive. When help was needed it 
should be in the form of a gentle reminder rather than a harsh instruction. 

2.1.6 Ease of Use 

Because many of the students were unfamiliar with computers it was important that 
the program be made as easy to use as possible. To this end keyboard skills were not 
necessary as all interaction would be done via the mouse. There would be a compre- 
hensive introduction and backup material. Interaction methods should be consistent. 

2.1.7 Flexibility 

The program needed to be flexible enough for the student to enter at his or her per- 
ceived level and exit or return to the start or previous level whenever he or she 
wished. The response method would be kept consistent to aid the ease of operating the 
program. 

2.1.8 Screen Layout 

Text would be large, brightly coloured and in upper and lower case as in normal read- 
ing material. A variety of screen colours would be used with feedback messages and 
instructions in different colour combinations. Illustrations would be kept to simple 
line drawings to avoid cluttering up the screen. The typeface and illustrations would 
be reproduced in the backup material but in black and white as dictated by economy. 

2.1.9 Technical requirements 

It was important to remember in the design of the program that it would run smoothly 
on the library computers. Tests would have to be made frequently on these computers 
as the program itself was being designed on a much higher grade machine. 

2.2 What was the program going to do? 

One of the most successful activities in the reading classes has been that of Language 
Experience where the students’ own experiences and conversation are used as a reading 
medium. Using this text in cloze exercises has resulted in the student being able to read 
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whole passages of text. Gee and Freebody have written that it is not enough to read, a student 
should read something meaningful. 

You can no more cut the literacy out of the overall social practice or cut away the non liter- 
acy parts from the literacy parts that you can abstract the white squares from a chess board 
and still have a chess board (Gee 1992) 

Literacy practices are written into the fabric of our social life ... Adults learning literacy are 
not learning what school children learn, and that, far from being an odd poor cousin, adult 
literacy is in fact forced to be literacy at work, culturally, economically and ideologically. 
(Freebody 1992) 

It was therefore decided to adapt the Language Experience approach and cloze exercises to a 
computer program using a multimedia authoring program. To make the program easier to manage 
it was decided to concentrate on the ‘lower’ end of ability, those students who were having the 
most difficulty in reading. It was also decided to use the students from one campus (Bowen) as 
‘guinea pigs’. The students at Cannonvale would be the control group. 

2.3 What words would be ‘taught’ 

The Language Experience approach had shown that what would appear to be the more difficult 
words such as ‘cabbages’, bananas’, ‘elephants’ or ‘screwdrivers’ were not the ones with which 
the students had the majority of problems. This is probably because they are usually identified 
with picture clues in reading material or by labels on the actual objects themselves. Difficulties 
occurred with the more common words such as ‘when’ ‘where’ ‘about’ or ‘their’. The resource 
Working on Reading, a practical approach to the teaching of reading’ contained a list of the 100 
most common words in English which included most of the words which were proving difficult. 
This was decided to be used as a starting point, more could be added later as was deemed neces- 
sary, using words identified by the students. 

2.4 What text would be used? 

It was decided to use material which meant something to the students, other computer packages 
have been found to be lacking in that they are too juvenile for adults or have material relating to 
North American or European lifestyles. Material reflecting tropical Queensland life and activities 
would be ideal. Class discussions with groups of students resulted in a series of story lines, each 
comprising a number of simple sentences. Each sentence contained some of the list of 100 words. 
These stories were relevant to the lifestyles and the spoken English of the client group, they also 
felt that they had ownership of the program. The sentences and storylines were graded to become 
progressively harder. 

2.5 Production details 

I undertook to design the package, illustrations were commissioned, some by the students them- 
selves and some from a local artist, and sound using real articulate Australian voices was 
arranged. Students were eager to be involved in this part of the program, usually shy people were 
volunteering to have their voices recorded by the sound studio on the computer. It was also a 
good lea rnin g experience as they were fascinated by the new technology of the colour scanner 
and programs such as CorelDRAW, Authorware Professional and Animator Pro. The project was 
having the effect of really making the students lose their fear of computers. 
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3. Description of the Program 

The program is based on the stories and sentences composed from the life experiences of 
all students although one group was involved more than the others as they were full-time 
students at the college. Each sentence contains at least three words from the list of 100 
most common words in English referred to previously. The program contains a number of 
‘stories’ or levels, each story is composed of a number of sentences. The tasks get progres- 
sively harder as the levels increase. 

At the start of the program there is a brief video sequence explaining how to use the pro- 
gram: including the use of the mouse, how to exit or return to the start or another level, 
selection of lessons and the advice to call a teacher or tutor if they felt that they needed 
help. 

All other interactions with the student appear in print form accompanied by a voice over, 
actions are controlled by the student using the mouse. 

The first screen asks the student to determine the speed of the program. 

The next screen asks the student to select the story or level required. 

The student is then asked to select a ‘lesson’ or sentence. 

At any time the student can quit, restart the program or go back one screen by using the 
mouse. 

The lesson starts with a blank coloured screen, each key word then appears separately in a 
contrasting colour on the screen along with a voice over saying the word. 

These words disappear and the lesson sentence appears in a different colour from the key 
words along with a line drawing illustration and a voice over reading the sentence. The sen- 
tence disappears leaving the line drawing. 

The sentence reappears with the key word to be learned in a different colour. This colour is 
the same as that of the key words shown previously. The voice over reads the sentence 
stressing the key word. 

The key word then disappears leaving a blank space. 

The key words then reappear in the same colour as before accompanied by a voice over. 

The student can use the mouse to click on the missing word and drag it into the sentence. 

If the wrong word is chosen the word will not be accepted into the sentence and the student 
urged to try again. 

The correct word will be accepted with positive feedback praising the student. 

All of these instructions are in print accompanied by voice overs. 

This sequence is repeated with different words in the sentence. 

At the end of each lesson or sentence a summary screen appears giving information on: key 
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words learned, words which were correct first time and words which required more attempts. 

The screen now returns to the lesson choice screen where the student can choose to retry the les- 
son or choose another sentence. Each lesson is accompanied by a different line drawing and a 
different colour combination of screen background, sentence print, key word print and instruction 
type print. 

The program was packaged and transferred to a number of computers so that more than one stu- 
dent could work on it at the same time 

To back up the program, print material was made of the lessons and each participating student 
was given a copy of this print material. This was composed of a booklet for each level or story 
containing a list of the key words, the same illustrations, the sentence written out in full on one 
page and the illustration and the sentence with the missing word on the next page. They were 
asked to write in the words in the spaces provided. Teachers could be approached to help if need- 
ed, but no specific teaching was given for the completion of this book, although time was 
allowed for the work during regular lessons. 

4. Trial of the program 

4.1 Pretesting 

In order to see if the program was having any effect some form of testing was necessary. This 
was explained to the students and they were given the option not to take part in the test or the 
trial as they wished. All volunteered to be part of the program. 

The students were all given a test of word recognition of the 100 most common words. This was 
done by first giving them the words in sentences and then asking them to read a word list where 
the key words were on their own. They then did a cloze exercise on the words. No time limit for 
completion of the test was set and no help was given. It is my concern that this testing was artifi- 
cial and related to Gee’s observation: 

... reading is what reading tests test ... (Gee 1992) 

However, it did give me a starting point and a point of reference on which to base my research. 
None of the students objected to the testing, even those who had had no part in the design of the 
program. 

4.2 Students involved 

Thirty students enrolled in basic literacy courses were asked if they would be part of this 
research. 

Three groups were involved, a daytime Aboriginal and Torres Strait Islander group, these were 
the ones who were involved in the design of the program, an evening community class and a day- 
time tutorial class. 

4.2.1 Aboriginal and Torres Strait Islander Group 

This group scored the best overall on the pretest, this could be because they were already 
familiar with the material. Scores for this group averaged at 85% accurate. 



Language Australia 



4.2.2 Evening Community Group 

The evening community class is of very mixed ability. Some had little difficulty with 
the words, others found them very difficult. Although the average score was 75% two 
students scored 100% and one third of the class scored less than 50%. One third of the 
tests were presented incomplete. 

4.23 Daytime Tutorial Class 

The daytime tutorial class has the most number of low level students. Three of them 
are Endeavour Foundation referrals who had great difficulty with the word recogni- 
tion. However great pains were taken to ensure that they did not think that they were 
failing anything. This class scored an average of 40%, over half the tests were pre- 
sented incomplete. 

4.2.4 Cannonvale Control groups 

These students are similar to the Bowen evening community class, a very mixed range 
of ability. Most are casual workers in the Hospitality Industry. They were given the 
words to read and to write down just the same as the Bowen group but were not told 
of the program at this stage. They just accepted it as a class exercise. This class scored 
an average of 70%. One third of the tests were returned incomplete. 

At the end of the pretest period the students were quite happy to commence working 
on the computer. 

4.3 Actual trial 

Each student was instructed individually on how to use the program. The time to do this 
varied greatly. Some were already fairly computer literate, especially the Aboriginal and 
Torres Strait Islander group, some had never touched a computer in their lives. All were 
enthusiastic and eager to take part in the program. They were then allocated time within 
their regular lessons to use the program and also informed that they could come and use the 
computer at other times during the day or even at specified times in the evening. It was not 
necessary for a teacher or tutor to be present when they were actually using the computer. 
However, in order for me to keep accurate records they should sign in and sign out of a log 
book so I could assess the time spent on the computer. 

Students were asked to give feedback on as many aspects as they could think of. They were 
told that any comment they made would be valued, even if they thought that the point was 
trivial. They were also told that they would be given a checklist at the end of the trial peri- 
od. 

The time for the trial was set for four weeks. 

4.4 Observations of students 

Students all tried the program out in the lessons. After the first lesson the time on the com- 
puter was made optional and if they wished other work could be set for them. 
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4.4.1 Aboriginal and Torres Strait Islander Group 

This group all competed the entire program and also the printed material. Most of the work 
was done in lesson time although two of the students became so fascinated that they 
returned in the evenings. One student brought in her husband and children so they could see 
what she was doing. All the family used the program treating it as a computer game. One 
student monopolised a computer every lunchtime until he had finished the program. He told 
me later that he was treating it as a personal challenge. Only one student had problems with 
the material, tutorial help was given to this student. None of these students had any prob- 
lems with the operation of the computer or the program. 

4.4.2 Evening Community class 

Two students decided that this was not for them as it was ‘too easy’ and so preferred to 
continue with their regular work. These were the ones who had scored 100% on the pretest. 
Of the others, the majority could cope with the content but all needed help with the comput- 
er operation. One student came in on another night during the week to use the program. 
Only half of this group finished the program and less than this number finished the work- 
book in the trial period. They did, however, express a desire to continue to work on this 
material after the trial period Three students in this group found the combination of learning 
a computer and also doing literacy work too much and did not continue with the program 
although they continued to attend class. 

4.4.3 Daytime Tutorial Class 

This group accepted the program with enthusiasm. They all spent as much time as possible 
on the computers, one person attended college on two days other than his tutorial days to 
use the program. Although each person was only required to attend class for one hour, in 
practice half of the students were spending all afternoon in the college working on the pro- 
gram or the workbook. Even with this amount of contact time at the end of the period only 
one person had completed all the program and none of the students had completed the 
workbook, but all were continuing with the project. Of this group most people needed quite 
a lot of help initially with the computers but once they got the idea they coped very well 
and most were able to work at their own pace without any tutor help. Two people still 
required tutor assistance all the time and this was given. 



5. After the trial 

After the four week period the students were given a repeat of the pretest and also a checklist 
checklist to complete. 

5.1 Results of the post test 

As with the pretest no publication was made of scores but all the students knew their own origi- 
nal score and were eager to see if they had improved. No time limit was set for completion of the 
test, and no help was given. 

5.1.1 Aboriginal and Torres Strait Islander Group 

The average score for this group rose to 98%, the majority of the group scored 100% on all 
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parts of the test. The ‘wrong’ answers originated from confusion over ‘there’ and 
‘their’ or ‘to’ and ‘too’. 

5.1.2 Evening Community Class 

The average score remained at 75%. This, however, was a general improvement as the 
two who had scored 100% in the pretest did not take part in the post test. More tests 
were returned complete than in the time of the pretest. 

5.13 Daytime Tutorial Class 

This group had by far the greatest increase in scores, rising to an overall 75% accura- 
cy. This probably relates to the fact that only two out of eight tests were returned 
incomplete. 

5.1.4 Cannonvale control group 

The average score increased to 80% with slightly more papers returned complete. 

5.2 Feedback from Checklist 

The checklist was designed to be as user friendly as possible and did not require the student 
to do any writing unless he or she wished. Students were asked a question and then asked to 
circle a response from a choice of between three and five options. Help was given to com- 
plete this questionnaire if needed. Twenty five students completed the checklist. Only 
Bowen students were given the checklist to complete. 

5.2.1 Difficulty of Computer program 

The choice was ‘too hard’ ‘fairly hard’ ‘OK’ and ‘easy’. 

Too hard 5 
Fairly hard 5 
OK 9 
Easy 6 

5.2.2 Difficulty of material 

The same choices were given. 

Too hard 5 
Fairly hard 3 
OK 7 
Easy 10 

5.23 Appropriateness of Material 

The choices were, ‘meant something to me’ or ‘meant little to me’ or ‘meant nothing 
to me’. 

Meant something to me 15 
Meant little to me 10 

Meant nothing to me 0 
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5.2.4 What was learned 

Did you leam, ‘a lot, ‘a little bit’ ‘nothing’ 

A lot 10 
A little bit 15 
Nothing 0 

5.2.5 Feedback from computer 

Students could chose as many of these words as they wished. 

Helpful 15 
Not helpful 8 
Friendly 8 
Unfriendly 5 
Rude 0 
Annoying 5 
Welcome 6 

53 Verbal Response to Teacher 

Each student was interviewed and asked their comments on the program. I was interested to 
know their suggestions on how the program could be improved and what changes I should make. 
In the course of the interviews the following emerged as the most common suggestions for 
improvement. 

53.1 Speed 

Although students were invited to adjust the speed of the program at the start they were 
only given three choices: slow, average or fast. Two students thought that ‘slow’ was not 
slow enough and would have liked the ability to halt the program for periods while they 
consulted the workbook. The majority thought that once they had become familiar with the 
program that ‘fast’ was not fast enough. It was recommended that at least two other speed 
levels should be introduced and that they be given numerical labels rather than the words 
‘slow’ or ‘fast’, especially ‘slow’ which was thought to be demeaning. 

53.2 Pretesting function on computer 

Some students would have liked the ability to test themselves on the key words of a lesson 
before attempting the lesson. They could then choose either to skip this lesson or use the 
pretest as a self evaluation tool when it was compared with the post lesson summary. 

533 Limitation of tries 

The program allows the student to keep trying indefinitely and will not leave the current 
task until the correct word has been identified. Students would have liked a limit of two 
tries to be imposed This would cut down the incidence of guesswork or of trying all the 
words in turn as you were sure to get the correct one by a process of elimination. It would 
also speed up the program. 
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53.4 Colour 

Some students felt bedazzled (their word) by the use of colour. The program has a dif- 
ferent colour combination for each lesson and they would have liked to see this cut 
down to three or four combinations in total. They felt that they would then be able to 
concentrate harder on the program. The use of red was to be discouraged as this was 
identified as being hard to pick out from a brightly coloured screen for some students. 
The line drawings for illustrations had originally been shown on a white background. 
This was changed to make the background the same colour as he screen. The students 
found this change easier on the eye. 

53.5 Feedback 

The feedback, although generally considered helpful and friendly, was thought to be 
boring and predictable. More variation was needed in the wording of the feedback. 
Strong support was voiced for the ability to print out the lists of words and the results 
screens at the end of each lesson. 

53.6 Voice overs 

A variety of voices had been used in the program. It was remarked that all the voices 
were warm, friendly and non threatening. Students, however, felt that we should stick 
to one voice and unanimously chose the clear tones and slow delivery of one of the 
library staff at the college. 

53.7 Content 

Students would like to continue with the program but would like the words and stories 
changed. What they really want is for a personalised program for each student. At the 
present time this is impossible because of the time it takes to write the programs, but 
extra stories and key words can be incorporated into the existing program. 

53.8 Accessibility 

Sometimes students had difficulty in accessing the program because all the computers 
were in use. It was recommended that we put the program on to more computers. 

Some students would have liked to put the program on to floppy disks so they could 
take their work home. This is possible if the space intensive voice overs are deleted 
and will be done. 

6. Behaviour changes 

As was mentioned at the start of this report I was interested to see if the introduction of the 
computer program had any effect on the motivation and irregular attendance of the students. 
There were some changes but it would be inpossible for me to say whether these changes 
were the result of the program or merely a Hawthorne effect, although students were not 
aware that I was considering these factors as part of the trial. 

6.1 Attendance 

Records of rolls kept before and during the trial show that attendance improved during the 
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trial for all groups. This was most noticeable in the Aboriginal and Torres Strait Islander group 
who went from an average of 80% attendance for the four weeks before the trial to 95% during 
the trial. There can be no record of attendance after the trial as the students are now on holiday. 

6.2 Motivation 

A log kept by students of computer use showed that some were coming in to college in their own 
time to work on the program. Certainly the computers were in full use each lunchtime. The after- 
noon tutorial group were arriving earlier and staying longer than their allocated time so they 
could use the computer. Some feedback I have received from the students indicates that the pro- 
gram needed to be put on to other computers to meet demand for its use. Observations of stu- 
dents showed that they appeared more enthusiastic, they asked lots of questions and generally 
showed an interest in the program. 

Some programs do provide desirable outcomes such as: independent, selfpaced learning, 
individual authoring of material, student interaction and improved editing and legibility. 

The status and interest attached to computers may also enhance the motivation to learn. 
(Taylor & Thommassen 1991) 

6.3 Self Esteem 

Observations of students have shown a rise in self esteem and also a rise in their status in the 
eyes of other students. Students on other courses were interested in what the literacy students 
were doing on computer and often engaged with them in conversation about the trial. The stu- 
dents who had had a major part in the designing of the package were eager to discuss their activi- 
ties. This was a breakthrough as previously the literacy groups had tended not to integrate with 
the other students. Some of the students engaged on the trial have asked me about careers in 
computing and have set themselves career goals in this area. 

7. Reflections 

Computers are now a way of life and if you do not know anything about them you are way 
behind in the social stakes. The cartoon Doonsbury featured on December 23 1995 in ‘The 
Australian’ illustrates this point when a seven year old child tells her father that she cannot hold 
up her head at school if her Christmas present computer is merely a Pentium 60 when her friends 
have machin es with a capacity of over 100. To meet the demands of an increasing computer lit- 
erate society we should be using this technology increasingly in our teaching strategies. 

Courses will need to incorporate technology in course design and delivery mode. Teachers 
will need to demonstrate awareness and willingness to apply principles of literacy and lan- 
guage in these technologies which are fundamental to open learning interactive multimedia, 
world wide web and Internet computer networks. (Kell 1995) 

Computers are also evolving rapidly and changing many of our preconceived ideas, threatening 
to bring even more drastic changes to our lifestyles. 

The rapid acceleration of technology also means that old methods of reading and learning 
will be replaced by new forms of reading and learning. (Kell 1995) 

My computer program sought to introduce computer technology into the teaching of basic key 
word recognition. The program was not perfect and students have given me many suggestions for 
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improvement which will be incorporated where possible. However, the students did 
improve their word recognition, they also developed more active motivation. Their status in 
the college grew with their own increased selfesteem and confidence. 

This is only the start. 



Maureen Mortimer 
Whitsunday College Barrier Reef Institute ofTAFE 
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WHO ’S LEARNING FROM WHOM: 

A NEW TEACHER EXPLORES LITERACY 
ASSESSMENT AND TEACHING 
STRATEGIES IN AN AD ULT URBAN 
CLASSROOM 

Cherie de Pinna 



ABSTRACT 

This chapter is a report of a study of literacy teaching/leaming and assessment practices in a 
classroom of adult urban students. The project developed from ideas generated at assessment 
forums coor dinat ed by Dr Joy Cumming for the National Language & Literacy Institute of 
Australia (NLLIA). It was undertaken in conjunction with Ms Jean Searle, Lecturer in Adult 
Education at Griffith University. Mr Alex Gamblin, Director, Southbank Institute of TAFE and 
Ms Trisha Evans, Associate Director of the Indigenous Peoples’ (Aust.) Unit supported this 
research project being conducted at Southbank TAFE’s Kangaroo Point campus. 

The project funding in part enabled me to travel to the Northern Territory to meet and talk with 
educators in Darwin and Batchelor. The generosity of these people is acknowledged. Ms Maree 
KJesch, Ms Helen Clarke, and in particular, Ms Pat Beattie from Batchelor College gave gener- 
ously of their time and experience to advise me on assessment, teaching strategies and text 
deconstruction. Ms Lenore Dembski and all the staff at the Northern Territory Department of 
Education’s Aboriginal Development Unit (ADU) deserve special mention for their hospitality 
and generosity. In particular, Ms Lenore Dembski, Mr Brian (BJ.) McGinness, Ms Lee Clark, Mr 
Kevin Parriman and Ms Barbara Smith took time to explain the training programs they are coor- 
dinating with Aboriginal communities in rural and remote areas of the Northern Territory. Ms 
Rae Flanigan from the Faculty of Aboriginal & Torres Strait Islander Studies (FATSIS) at the 
Northern Territory University highlighted the importance of cultural identity in teaching. 

This chapter comprises extracts from and information based on extensive notes taken throughout 
the semester, a contribution from some of the students, and the verbatim responses to questions 
posed by the author, about class activities. 

BACKGROUND 

The Access program at the Indigenous Peoples’ (Aust.) Unit (IPU) at Southbank Institute of 
TAFE comprises three levels - CN0016 Certificate in Basic Vocational & Educational Access 
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(ATSI), CN0017 Certificate in Intermediate Vocational & Educational Access (ATSI) and 
CN0018 Certificate in Advanced Vocational & Educational Access (ATSI). Students study 
literacy, numeracy, cultural studies and computers. 

My appointment at the IPU was on a casual basis. In first semester, I was a tutor with the 
Intermediate and Basic Literacy classes, but in second semester I was promoted to being the 
teacher of the new intake of Intermediate Literacy students. The classes were held for a 
total of six hours per week, divided into two three-hour sessions on the one day. This situa- 
tion was due to my own time restrictions. 

Nearly all the students in the class had negative school experiences and were retu rnin g to 
education after periods away from school. 

The class comprised 16 Aboriginal and Torres Strait Islander students from various parts of 
Australia, but mostly from Queensland and New South Wales. Ages ranged from 15 to 50+ 
years. Many of the mature-aged students were looking to change vocation and enter fields 
such as welfare, administration, and the police force. Some were wanting to go on to uni- 
versity. The students were part of the mid-year intake, so had to go into either Basic or 
Intermediate levels due to administrative constraints. Many could have entered Advanced 
level, if this had been open to them. 

The curriculum modules used were LIT1 13 Literacy and Language Personal Focus - Level 
2 (1994), and LIT123 Literacy and Language Vocational Focus - Level 2 (1994). These are 
part of the Certificate in Vocational Access (CNL03). 

I was advised that twelve learning outcomes had to be achieved by the end of semester, 
based on the tables set out in the Content section of the two Module Descriptors. Each table 
has six headings - factual, persuasive, transactional, procedural, narrative and conversation - 
under which are listed several examples of each text type. My understanding was that I was 
restricted to these examples, which I found very limiting and frustrating. I wanted to take a 
broader view of lesson planning - a large semester-long project, for example, where literacy 
activities were carried out in context - but what I was presented with, together with my own 
inhibitions as a new teacher, stifled this inclination at times. I did not want to have lessons 
where texts were taught on a weekly basis, in isolation from their context, but this seemed 
to be what was requested of me. Largely I ignored this and followed my intuition, but there 
were times when I lacked confidence and wondered if I was teaching the “right” way. 

I also could not make sense of the Module Descriptors. The Learning Outcomes seemed 
vague, yet the Assessment Criteria referred specifically to only one text example in the 
Content table. It was only towards the end of the semester that I understood the flexibility 
offered in these Learning Outcomes. 

PRINCIPLES OF LEARNING INTERACTIONS 

The word ‘teacher’ is in my view a misnomer with know-all connotations. I wanted to test 
my belief that no one person has the monopoly on knowledge and that everyone is a teacher 
and a learner, engaged in the process of swapping knowledge. I was curious to find out if 
students leam best when they are confident that their teacher respects them, and values the 
experience and skills they bring to the classroom (Shore et al., 1993) . 

I was also interested to test my theory that constructive feedback, rather than summative 



111 



Adult Literacy and Numeracy: Assessing Change 



assessment is conducive to learning and a positive learner image. Students who have had nega- 
tive school experiences need the encouragement of non-threatening assessment while they build 
up their motivation and confidence in their abilities (Wilson, 1995). 

My belief that students respond better to criterion rather than norm-referenced assessment, and 
that their commitment and involvement deepens as they take on more responsibility for their 
learning needed to be examined. I also felt that norm-referenced assessment, with its competitive 
nature, did not foster the cohesion and team spirit that I was hoping to achieve in the classroom. 
Also, Aboriginal and Torres Strait Islander peoples place importance on cooperative and collabo- 
rative learning, and although this would not apply to all students, it seemed appropriate in this 
instance. 

RESEARCH METHODOLOGY 

The major focus of the study was to be an account of my exploration of assessment activities in 
the integrated learner-focused classroom that I envisaged. Approval was sought from TAFE and 
the IPU management to carry out the research during second semester, 1995. The students were 
informed of my research project and grant, but it was not until the end of the semester that I 
asked for their feedback. This was because as my research progressed and revealed other issues, I 
was unsure at times, just what the most important aspect of my research would be. I wanted to be 
absolutely clear in my mind what I wanted to ask the students before I approached them. If I was 
not clear, I felt that my research would appear an intrusion rather than something we could work 
on together. When we did discuss it, we decided that the students would contribute a section 
about a particular class activity, and that we would hold an informal feedback/evaluation session 
about the course (see Appendix 1). 

It has been my practice since I started teaching to write detailed notes about the lesson and each 
student’s progress after every class. These notes include facts, observations, “... ideas, insights 
and feelings” (Searle 1991). I do this in order to crystallise thoughts, clarify points and plan the 
next lesson. 

Collecting data for the study, therefore, was an extension of this practice. My notes comprised 
detailed note-taking about each student and the class, plus three columns outlining what I had 
planned, what had/had not worked, and why. 

I also read extensively about Aboriginal education and Aboriginal and Torres Strait Islander cul- 
ture, held informal discussions with Aboriginal and Islander colleagues, friends and students and 
absorbed everything I was learning from the students and others around me (Spradley, 1979). 
Christie, Walton and Eggington, Harvey and McGinty, Petman and Dixon were some of the 
authors and editors who informed my thinking (Shore, 1991) but in particular, it was the students 
and the class tutor from whom I learnt the most. 

At the end of the semester, the students and I held an informal feedback session based on ques- 
tions compiled by me. The questions dealt mainly with assessment and teaching strategies, 
though we did discuss other activities that had occurred during the semester. Verbatim answers 
were recorded in shorthand by me. Time constraints and student absences on this particular after- 
noon, meant that only ten of the sixteen students who normally attended participated in the dis- 
cussion. This was not altogether a negative situation, as more than half of those present were nor- 
mally more reserved students, who willingly expressed their opinions. 

Analysis of the data proved the most troublesome part of the research. I was frustrated by the 




105 



Language Australia 



amount of data I had collected, and felt lost as to where to start (Searle 1991). After seeking 
advice, I sorted the data according to assessment and teaching strategies that did or did not 
work. The report was then written as a narrative. 

REFLECTIONS ON THE COURSE 

When I found out that I was to have a class of my own I decided I wanted to develop a 
close, secure environment, which was conducive to learning. I envisaged a culture operating 
within the classroom, in which students felt responsible for and committed to their learning. 
A key part of my role, I believed, was to make the classroom as interesting a place as possi- 
ble, so students would want to come - a contrast to their school experiences. 

I wanted the students to feel confident about their ability to carry out the skills they learnt, 
and I felt this could only be achieved in a supportive atmosphere. I also wanted the students 
to change their perceptions of learning. I hoped that they would discover its joy, and see 
themselves as learners whose opinions are sought and respected. 

ASSESSMENT METHOD 

It is commonly accepted that good assessment practice is criterion rather than norm-refer- 
enced. I felt it was necessary also for me to relinquish the power that teachers have to stand 
in judgement of students’ work (Shore et al., 1993). I wanted this power to be handed over 
to the students as much as possible, so as to build on their confidence and image as learn- 
ers. It was just as important to involve the students in the assessment procedure as it was 
any other part of their learning (Osmond, 1990). 

The criteria for the learning outcomes was decided collaboratively, with significant contri- 
butions by people who had experience in the text styles or activities (e.g. meeting proce- 
dure) we were studying. As students had more experience than I in some areas, it seemed 
logical that this knowledge should be tapped. 

Text styles were modelled and scaffolded, always with student participation. Depending on 
the level of difficulty, the students then either worked in groups or attempted the task inde- 
pendently straightaway. All work, including the modelling and scaffolding, was written into 
their journals. Both verbal and written learning outcomes were assessed against the criteria 
upon which we, as a class, had decided. Assessment of text styles usually took the form of 
my constructive comments written in the journals, which were collected at the end of the 
lesson, then returned the following week. These journals had been distributed at the begin- 
ning of the semester. Verbal tasks, such as speaking in public, were peer-assessed. 

Summative assessment was not used at all. In my view, tests and grading are detrimental to 
people who see themselves as poor learners. This assessment method has its place, but only 
when students are more self-assured about their abilities. 

EXTRACTS FROM CLASS NOTES 

The following extracts taken from my notes highlight issues that I felt were important in 
my development as a teacher and the learning of the students. 

31 July 1995 

On this day, about 10 new students joined the class. In fact, it was not until the third week 
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that the numbers stabilised and I knew for sure who would be in the class. This created some dif- 
ficulties, as the beginning group had worked out rules and negotiated some learning outcomes. It 
also meant that I was constantly introducing class members to new people, but eventually we 
decided to dispense with this, and the students would say their names themselves. On the third 
week 24 students were in the class, though this was to reduce to a steady group of 13-16 for the 
rest of the semester. 

When each new member arrived I would ask them, informally, about their personal goals and 
ambitions as I wanted to establish a rapport quickly. It was important that they knew I was inter- 
ested in them as individuals. In this third week, having expressed their career goals, we decided 
on the lear nin g outcomes outlined in the modules that would most benefit them in the future. 
Some of the late arrivals had already completed some of the outcomes with the Basic level 
teacher, so they were able to catch up easily. 

The learning outcome we worked on at this stage was meeting procedure from Learning 
Outcome 2 which states, “Use literacy and language texts related to participation, discussion and 
decision making with class groups, e.g. read and act on minutes of meetings, take personal notes, 
negotiation, rules” from the Vocational Focus module. 

I explained to the students that it seemed pointless to have a meeting without a purpose, so we 
decided to do it within a context. We discussed various topics, but when someone suggested 
fund-raising for an end-of-year party, we all agreed. We then debated what our first venture 
would be, and after listing about 10 suggestions on the board (ranging from stalls to basketball 
competitions) we decided on a raffle - a meat raffle. During this lesson, we elected a chairperson, 
secretary, vice-chair and treasurer for future meetings (though this was later to change to turn- 
about). About three students had extensive experience on Aboriginal community boards and com- 
mittees, so I encouraged them to speak and share their knowledge, which they were happy to do. 

At this class, journals were also distributed. When I was tutoring last semester, I had noticed that 
students were forever asking for distributed notes because they had either lost them or missed the 
class. The teacher kept having to use her morning teas and class time for re-photocopying materi- 
al she had distributed the week or two before. I was determined not to do this, so gained approval 
from the IPU to purchase 25 exercise books (which were on special at 49c each) for students to 
use as their literacy journals. All the students had to bring every Monday was this exercise book 
and a pencil, and they would have everything they needed. I wanted to simplify things, to give 
them the best opportunity to be organised, so as to build on their self-image as learners. 

Someone suggested that as we had an exercise book left over, we use it as a minutes book for our 
meetings, which we agreed was a great idea. 

28 August 1995 

The class began with organising the raffle. Students divided the raffle tickets into books of 10, 
while others put the agenda together for the morning’s meeting. It was on this day that the meet- 
ing began to take on the role of an information forum, with students suggesting items other than 
fund-raising. Some of the items for discussion were tutors, culture studies trip etc. I felt this was 
a great step forward, as the students took on the responsibility of the meeting. Developments in 
student interaction and group trust emerged during this class. In order to analyse the text, we 
moved the desks together, so we could all sit around a table. 

In this lesson, I attempted text deconstruction based on Pat Beattie’s “Teaching Notes from the 
Special Teaching Unit - No 3. Deconstructing a print text with students” published through 
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Batchelor College. Inexperience and lack of confidence meant that I failed to encourage the 
students to “dig deeper” (Beattie, 1995) into the text and critically analyse it. However, 
though we remained at a superficial level regarding text deconstruction, we worked on con- 
necting words, which was a problem area for some of the students. 

In the afternoon, we looked at note-taking from texts and practised looking up tables of 
contents, indexes etc. We then went to the library for library search tuition and to put what 
we had learnt in the classroom into practice. 

9 October, 1995 

Over the weeks, indications of students taking control of their learning emerged. One of 
these indicators was the desk arrangements. Every lesson, when I arrived, I found the desks 
already arranged in a U-shape, ready for the meeting. This day was no exception. 

During the week, I had decided that, in light of earlier classes, the students needed to learn 
how to write a letter of complaint that provided constructive feedback. I explained why I 
thought this was important. Together, we scaffolded a letter on the board. Students offered 
suggestions for changing it and making it more succinct. We considered, wrote and erased, 
considered, wrote and erased. I explained that by doing this we were experiencing the writ- 
ing process, in that we were polishing our work as authors do. Everyone was very happy 
with our finished product, and they said that they would appreciate receiving a letter like 
the one we’d written. They felt it was constructive and positive, while relaying a sincere 
message. One student said it showed the writer was assertive. This was a good lesson. 

The students then wrote a letter of their own on a topic of their choice. Some were fluent, 
concise and well-considered letters about issues such as pollution, French nuclear testing 
and wheel chair access in Brisbane’s Queen Street mall. Two students wrote about them- 
selves - what they had been like and how they could improve. We read these out to each 
other. 

One of the students raised some concerns about completing all the learning outcomes in 
time for graduation. I saw this as an opportunity to discuss learning outcomes in broader 
terms, in the hope that students would see learning as more than fulfilling the competency 
criteria set for particular tasks. 

ASSESSMENT 

The following extracts detail specific assessment activities which took place during the 
semester and the reactions of the students and myself to their implementation. 

18 August 1995 

I assessed the students’ resumes and wrote copious notes on their work. At this time, I read 
“Collaborative revision: learning and self-assessment” by Michael C Flanigan who argues 
that students do not absorb the notes written by teachers on their assignments, because they 
are not involved in the assessment process. When I thought back on my own experience, 
this made sense to me, so I rubbed everything out and started again. Instead, I gave hints so 
the students would have to find the mistakes. For example, the notes read, “Campbell, This 
is a neat and concise resume, however there are two typing errors on page 1 and three on 
page 2. When you’ve corrected them, please give me a copy of the resume’ for my files. 
Thanks, Cherie”. This way, I hoped that students would discover their grammatical and 
spelling mistakes and be motivated to correct them. 
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I imagined students, inspired and enthused by my notes, analysing their work, then poring over 
books eager to find the answers to the posed questions. Instead, I found it was some time before 
the corrected resumes came back to me - if, indeed, they did at all. I couldn’t work out why this 
had happened until I spoke to another literacy teacher. She explained that as well as giving hints I 
should have helped the students identify how to correct their work. How would a poor speller, for 
example, be able to identify a spelling mistake? This had crossed my mind, but I had dismissed 
it. It was up to me to discuss possible solutions with the student. 

4 September 1995 (extract) 

The fact that we had organised a raffle provided an ideal opportunity to introduce report writing 
and thereby write about the raffle. 

... After a short discussion and explanation about report layouts, and scaffolding brief sen- 
tence examples, I asked the students to brainstorm in small groups then write individual 
reports. Everyone was unsure as to what to do, as my request was met with blank stares and 
confused looks. What was the problem? Had I been unclear in my explanation? After talk- 
ing together, I realised I had to scaffold in more detail. We analysed and thought back on 
what we had done to organise the raffle, then wrote some of this information, in full sen- 
tences on the board. The sentence examples provided a basis for everyone to get started, 
and once this was worked out, they sat down together and wrote reports. The quality of the 
reports reflected the importance of thorough scaffolding. They were cohesive and well- 
structured, displayed a concise and clear interpretation of events, and were informative and 
sequential in their approach. 

Later, we read out some of the reports, and discussed the problems or otherwise of writing 
them. I realised that students have an unrealistic expectation of writing they think they 
should be able to sit down and write perfectly and that it should just flow from the pen. I 
explained that no writer does this and that writing, even for professionals, is very hard. 

9 October, 1995 (extract) 

One of the students asked about completion of learning outcomes. This question provided 
an oppor tuni ty to broaden students’ perceptions of learning outcomes from that of a nar- 
rowly defined task. After lunch, the class tutor and I devised a question for the class to con- 
sider. We wrote on the board - “Are the learning outcomes all the requirements you need to 
get a job, hold down a job and to operate effectively in both societies?” 

The students are fully aware of the need to have other skills and qualities in order to gain 
and remain in employment, and they know what these skills are. Some of the suggestions 
they made were: self-respect and respect for others, self-esteem, positive attitude, punctuali- 
ty, hygiene, attendance, dress, good manners. The topic expanded to a discussion on self- 
abuse, the effect this has on one’s work and personal life, and steps one can take to counter 
it. Self-evaluation, choice of friends and a change in attitude were some of the points raised. 

The tutor and I then asked the students to rate themselves on a scale of 1-10 against the cri- 
teria they had suggested. We asked them to be honest with themselves in their assessment. 
We then included the activity, “speaking in front of a group” in this session and asked for 
them to say their scores and why. It was interesting to note that most people tended to be 
modest in their estimations. The tutor and I though this session went really well, although 
later I was told that a couple of the boys had thought ‘I was trying to get them to “open up” 
tome’. 
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DISCUSSION 

My aim at the beginning of the course was to develop a strong, supportive and close unit so 
students would want to attend class, and feel motivated to leam. This was largely achieved, 
but it would not have been possible without the shared philosophy and aims of the class 
tutor, nor of the class members themselves. 

The journals proved a successful tool in assisting both myself and the students to assess 
their Learning Outcome competencies. As all the work done during the lessons was written 
in their exercise books, they had portfolios to which they could refer and see their progress. 

I have left it up to the students to say what they thought of the class (see Appendix). I have 
categorised what I learnt into two sections -professional and personal. 

Professional 

From the outset, I respected the experiences and knowledge each student brought to the 
class, and became part of the team, as student, coordinator and guide. It is comforting, 
therefore, to know that my humanist/radical educational philosophy is put into practice. The 
students and I shared information, and benefited from each other’s knowledge. Honesty and 
openness were an integral part of this. 

Decision-making and power was devolved as much as possible (Watson et al., 1981) 
although I always felt restricted by my inexperience. I was frustrated by my lack of knowl- 
edge as to how to hand over the power to the extent to which I wanted. If we had worked 
together for another semester, I would have been more confident and expanded the devolu- 
tion to include, amongst other things, more student involvement in assessment. 

This method of inclusive teaching is not popular with everyone, particularly with adults 
returning to education. Although I did not receive any complaints, I did hear that some of 
the older students would have preferred for me to have had a more conventional “chalk and 
talk” teaching style. 

I was also confronted by less attractive aspects of my teaching. For example, I noticed that, 
despite handing over the power to the students, I would try to take over or direct the situa- 
tion when I thought they were not making decisions quickly enough. I also had to watch 
constantly that I did not talk all the time. I would pose a question, then want to answer it, 
rather than wait and listen to forthcoming answers. I decided that I am afflicted with the 
“love to hear my own voice” syndrome, as it took all my will and self-control not to keep 
talking. 

My explanations are not always clear. I tend to be too wordy, and start a sentence too often, 
rather than thinking before speaking. When I suggested this to the students, some of them 
agreed. Others, I suspect, were too polite to say so. 

There is also a strong desire in me to be “a role model of the exemplary teacher” 
(Brookfield, 1990). This is physically, mentally and emotionally draining and ultimately 
counter-productive. 

Personal 

Working with the Intermediate Literacy Class had a significant impact on my life. While all 



O 

ERIC 



110 



Adult Literacy and Numeracy: Assessing Change 



of us were students in the class, what I learnt had a profound effect on my thinking and attitudes. 
As I came to know the students, I questioned my assumptions about my place and importance in 
the world. I questioned my values and lifestyle and decided to make some long-overdue changes. 

I also gained a greater knowledge and understanding of Aboriginal issues and the effect white 
settlement had and continues to have on Aboriginal lives and culture. 



Cherie de Pinna 
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Appendices 

STUDENT REPORT 

Meat Raffle Report - 4/9/1995 

by 

Jack Williams 
with contributions from 

Ernie Bond, Carmel Burke, Richie Charles, Donna Newley, 
Belinda Ross, Jimmy Smith (tutor), Henry Thompson, 
Robert Turnbull and Dana White 1 



Background 

The Intermediate Literacy Group, from the Indigenous Peoples’ Unit, formed a social club 
which they called the Intermediate Social Club (I SC). The social club decided to hold a 
raffle. It was decided to have a meat tray. The money from the raffle is for a get-together at 
the end of the year, as this is the easiest way to make money. 

Discussion 

The class nominated a committee consisting of a chairperson, vice-chairperson, secretary, 
treasurer and Board of Committee. The ISC voted that each member contribute two dollars 
each to start the raffle off, from which the treasurer bought the two books of tickets, which 
contained one hundred tickets each. It was decided to sell each ticket for one dollar. 

The treasurer opened a bank account to put in the profits after expenses. The social club 
bought a meat tray for S20; the club now has S72.50 in the bank. One of the members had 
an idea of making up some posters for advertisements to tell other students of our raffle. 

The organisation was first thought of on the 7th August. The raffle was to be drawn on the 
1st September, but as all the tickets were not sold, it was decided to draw it on 7th 
September. The raffle was run only at the Indigenous Peoples’ Unit at the Kangaroo Point 
College. 

Recommendations 

(i) It is recommended that the people who are selling tickets bring them in on the due 
date. 

(ii) It is recommended that the names of all the people who took books to sell be record- 
ed, so it can be known which books are outs tandin g 

(iii) The prize will be on display, if it is suitable. 

(iv) The ISC is going to have fewer tickets for sale, which might mean a rise in the price 
of each ticket. 

Conclusion 

The ISC could have organised the raffle a lot better. There were a few books of tickets out- 
standing. It started with $23.50, the meat tray cost $20.00, which means the ISC has sold 
seven books at ten dollars a book. The total now in the bank is $72.00. 
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Students’ Comments About Raffle 

We thought that the first raffle was successful because we made some money for the end-of-year 
“bash”. We worked very well together to make it a success. One of the members made a poster 
to put up on noticeboards around the College. We decided we would run a raffle again, with a 
larger prize. We also thought the raffle was a success because we made a profit. 



Note: 

1 . All the authors mentioned by name agreed to have the report published and their names includ- 
ed. 
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STUDENTS* ASSESSMENT OF CLASS ACTIVITIES 

Below are students’ responses to a series of questions raised by me when evaluating the 
course. This was a round-table discussion at which I took replies in shorthand. Ten students 
were present. 

• What did you think of the physical layout of the classroom, i.e. desk arrangements ? 

(The desks were set up in a square) 

Excellent. It gives a round table effect. It means there’s good eye contact. People change 
rooms according to how they want them. 

Better this way than all in lines in rows. Communication runs more smoothly, because it is 
like this. It’s all open. There’s a more open feeling. 

It depends on where the teacher’s comfortable sitting. It’s better for seeing the teacher, 
because none is in the way. It’s better than looking at the back of peoples’ heads, and they 
don’t have to turn around and talk to you. 

It made us feel as if we mattered. We were more in a group. You can read the teacher’s 
face. You can look at the faces of the others and see what they’re thinking. 

It breaks down the barriers. We didn’t care where we sat. When I first arrived, I felt com- 
fortable sitting anywhere. 

It’s a good way to relate to Murri people. It makes people feel they’re connected. 

I’m happier looking at people, whereas Murris tend not to do that. It’s more conducive to 
looking at people in the eye. 

• What were the advantages/disadvantages of the meetings? 

Advantages: 

Experience 

The class got to voice opinions. We got experience in being a secretary and taking notes. 
We’re going to have to do it one day, and we’ll know what to do. We’re learning meeting 
procedures. We flow through the meetings now. It means we feel more confident within the 
jobs we have to do. 

Because it’s a classroom thing, and being Murri, we’re a family. We’re more than just a 
classroom of Murri people. 

“Brothers and sisters”. It’s a feeling of “oneness”. A particular group of Murri people is 
going to be a family. 

We talked about other issues other than the raffle. 

Disadvantages: 

None. 

There’s no disadvantages in learning something different. 

There weren’t enough people putting in their input, but this is personal. 

• Were the journals /exercise books useful? 

Good 

It means we were able to look at what we learnt. 

We could look at mistakes. 

I’d recommend we keep a journal. 

People aren’t handing in bits of paper. 

Tops (thumb raised). 

You can keep notes together. We can remember what we’re doing. 
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Tops (thumb raised). 

You can keep notes together. We can remember what we’re doing. 

Splendid idea. It met the requirement that we needed, it kept everything together. 

• What did you think of the after lunch exercises? 

We didn’t have them for long enough (they only had two sessions). 

It’d be good to continue them, but allow us time to settle in after lunch. 

We like the idea of having exercises. 

• What did you think of the class making up rules? 

Good. Good idea. 

It encourages confidence and self-esteem because you’re able to relax. It encourages people 
to talk. Everyone tried to encourage ... (a student). It’s a genuine caring for each other. We’re 
helping one another. It’s helping us move forward. We encouraged people to find confidence 
in themselves. We made the rules not to intimidate but to encourage people. Murris have a 
tendency to jeer. “Don’t laugh at people and respect others” were important points (in the 
rules). These are important to break the mould. 

They made us all equal. 

We feel more comfortable because we’re at home and can come out of our shell. 

You’re doing something yourself and not following rules set down by TAFE, but making our 
own meant we respect them a lot more because they come from our own decision-making. 

This means people are going to learn. 

• If another group was doing this subject, are there other things you would 

cover? 

Do shorthand as an elective. 

Typing 

Thesaurus 

Verbal communication 

Role play, like the interviews. We need to know how to relate to people at different levels. We 
need confidence building and self-esteem. We need activities to overcome feelings of inferiority 
and how to cope when they’re intimidating you. We need to learn more about stopping the effect 
that people have on us. We need to take control of the conversation. We need more role plays on 
how not to let people intimidate you. (This was during a discussion about how government 
employees intimidate the students and talk to them very badly.) 

• General feedback and comments were offered rather than grading. What did you think of 

that? 

Good 

Grading makes us feel a name in the book and not personal communication. 

If you’re talking in general comments, we get more out of it. 

You learn more. You get on top of problems. You’re more acceptable to be open to criticism 
than given a grade in a book with a mark. There’s a grading system within the unit, in that stu- 
dents are divided into Basic, Intermediate and Advanced. 

I prefer it (the comments rather than the grading). It makes me feel better about things. 

Your comments make me feel better. You wrote, “Your writing is so neat it’s a pleasure to read.” 
This made me feel good. 

It’s more acceptable to us. Positive comments make people feel good. You let us know when 
we’ve done something right, which means we’re more open to constructive criticism. 

No students would be here if you had us sitting in rows, grading our work and talking to us. 
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You get no feedback from the teacher and then at the end of the semester you get a mark 
and a comment in the book. I throw it away. It doesn’t mean anything to me. 

• What did you think of the outings? 

Great idea. 

It took us out of the classroom. 

We still learnt things. 

It’s good for class morale. 
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ARE LITERACY AND NUMERACY 
COMPETENCIES ACHIEVED BY 
NEWSTART CLIENTS TRANSFERRED TO 
THE CLIENTS’ REAL-LIFE CONTEXT? 

Maureen Cooper and Susan Garside 

Introduction 

In September 1995, the Literacy Language Centre of the Bayside Campus of the Moreton 
Institute of TAFE applied for and gained funding to conduct research into the transfer of literacy 
and numeracy competencies from the college learning environment to clients’ real-life contexts. 
The study was undertaken in a “Newstart” course at the Bayside Campus of the Moreton Institute 
of TAFE. Newstart is a full-time [200 hour] course for those unemployed people referred by the 
CES who need, in particular, to improve their literacy and numeracy skills. It is an access course 
which also includes computing, industrial knowledge, personal development and can be linked to 
other vocational courses. It might also be known in other contexts as a Special Intervention or 
Labour Market program. 

The aim of the project was to determine whether competencies achieved during the Newstart pro- 
gram were transferred to other contexts. 

We conducted this research project by asking questions in an informal interview situation of: 

1. Students who had completed at least one course 

2. Teachers of the course 

3. Employers, supervisors and a CES Case Manager. 

Students’ questions related to the use of the competencies that they had achieved by the end of 
the course. They were asked to describe incidents where the knowledge, skills and attitude 
learnt on the course were, or were not, put into practice successfully. Were they able to transfer 
their competencies from the learning environment to their real-life contexts of work or communi- 
ty? 

Teachers, employers, supervisors and the CES case manager were asked to talk about the culture 
of their workplaces and expectations they had of students/employees. They were further asked to 
comment on knowledge, skills and attitudes specifically related to the learning outcomes of the 
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Other anticipated outcomes of the project were information about: 

The reasons for lack of transference if applicable 

The clients’ perception of their ability to use their stated competence 

Clients’ thoughts on competency based training and the curriculum used 

for the Newstart course [Certificate in Vocational Access, CNL03 - Employment 

focus]. 

We also considered the skills that are needed to enhance transfer. 

The research was prompted by practitioners’ questions. Teachers assessing literacy and 
numeracy competencies using the CNL03 curriculum wanted to know if the competencies 
were being transferred to the clients’ real-life contexts. Was their rating of students as 
competent applicable to another setting? These questions suggest that teachers doubt that 
students can always transfer stated competencies to another context and they feel that this 
inpacts on their credibility as teachers. If students tell employers that they have been 
assessed as competent at a certain task but cannot perform the task in the employer context, 
what does it mean to be assessed as competent after a course? 

The CNL03 curriculum, accredited in February 1995, uses competency based assessment. 

In the Employment focus there are six literacy and six numeracy modules at progressive 
levels. In addition, there is one module covering workplace awareness, which includes 
workplace health and safety, understanding workplace change, award restructuring, the use 
of technology and human resource issues. Each literacy module has six learning outcomes 
[LO] which reflect the six aspects of the National Framework of Adult English Language. 
Literacy and Numeracy Competence. These six aspects are: personal, co-operative, sys- 
tems, community, procedural and technological communication. 

At whatever level the student is working, the first learning outcome of the Literacy modules 
concerns the students’ use of literacy texts in relation to their own personal identify for 
example, identifying and recording personal goals, writing personal details on forms. 
Learning outcome 2 relates to co-operative communication in the form of group interaction 
skills. This might include an oral recount to a group, receiving instructions from a supervi- 
sor or being an active participant in a team situation. The use of texts specific to organisa- 
tiQRSZsy Stems such as the workplace or the CES office are the focus of learning outcome 3. 
Relevant activities are reading and responding to job cards, safety instructions at work or an 
internal memo. The fourth learning outcome relates to the students’ interaction with the 
community, in such situations as, taking phone messages, asking questions or giving a vote 
of thanks at a meeting or reading and responding to an article in the local paper. The use of 
procedural texts is the focus of learning outcome 5 and might include, following safety 
instructions, operating a machine by reading a manual or explaining the procedure to com- 
plete a particular task. Finally, learning outcome 6 is about the use of technology whether 
it be a fax machine, computer or the telephone. 

In the numeracy modules, outcome 1 covers mathematical operations used in everyday situ- 
ations. The second outcome incorporates measurement in height weight, capacity, length 
and time and the third outcome relates to shape and position . The latter covers location of a 
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book in a library or getting to a bus stop. The final learning outcome [which is not included in 
the first module] concerns the graphic representation of data and might include reading a street 
directory or identifying the correct dosage of medicine. 

With the implementation of the CNL03 curriculum teachers feel pressured to work towards stu- 
dents achieving competence in the 6 learning outcomes in each literacy module and in the 3 or 4 
learning outcomes in each numeracy module. They feel that, in working toward a successful 
completion of the assessment tasks, they do not have time to ensure that students have sufficient 
practice to achieve a genuine understanding of the concepts involved in each learning outcome. 

Teachers are aware of the time that needs to be spent working with students in the affective 
domain in order to develop confidence to carry out tasks but they feel that they do not have the 
time to do this. They know that competence in a skill in a familiar supportive environment is 
quite different from being able to perform the same skill in an unfamiliar, less supportive envi- 
ronment and the ability to transfer learning is often associated with confidence [A CAL 1992], 

The pressure for teachers does not come from the Adult Literacy field which has always recog- 
nised [and specifically campaigned] that adults take a long time to achieve literacy competence at 
their desired level because of their lack of self esteem and low self-confidence. Nor does it come 
from the curriculum, which is in CBT format, and therefore operating on the principle that com- 
petencies can be achieved in whatever time it takes, two weeks or two years. The pressure 
comes from an administrative source, CAPS [College Administration Programs], which has to 
record specific learning outcomes against courses. This reflects a system where evaluation of 
programs is tied to successful course completion, that is, achievement of competence in all learn- 
ing outcomes, in the nominal time specified. However, successful course completion in access 
courses may well mean that the student stayed in the course for the duration but achieved no 
specified competencies. Further, the student may say that the course changed their whole attitude 
to employment and training and yet, in not achieving competence in all 6 learning outcomes they 
are deemed “not yet competent” and by present criteria not having successfully completed the 
course. 

Research on transfer of skills 

Human behaviour is significantly influenced by the settings in which it occurs and it cannot be 
assumed that competencies acquired in one context will be transferred to another. 

Billet [1994a] writes that; 

The provision of learning experiences that promote robust and transferable knowledge and 
skills is an enduring goal of educational practice. How to secure this goal remains the 
source of much debate, research and theorising. [p4] 

Various factors influence the successful transfer of skills and context is of major importance. 
Literacy is significantly influenced by the settings in which it is used as it is socially constructed 
and specific to each context in which it is found [Luke 1988, Wickert 1991]. This means that the 
context of the college will generate certain literate behaviours which may, or may not, be appro- 
priate for the context of work, home or job-seeking. Diehl and Mikulecky [1980] have found on 
worksites that on-the-job, extra-linguistic cues help workers to achieve more successfully than 
their general “reading ability” would suggest, demonstrating that literacy is embedded in the context of 
the work situation. 
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Mikulecky and Strange [1986] quote Sticht who comments that the US Army discontinued 
general literacy training to focus on specific job literacy because of the low transferability 
of general reading skills to specific literacy tasks in the military. Mikulecky and Strange 
[ibid] found that programmes they studied which addressed specific job literacy improved 
trainees’ abilities in these areas, but that there was little transfer to general reading ability, 
unless the participants did a lot of general reading at home. Successful work based literacy 
programs used job simulation as the major training device, they had high time-on-task and 
participants had individual attention and feedback. 

In relation to competency based training, Thompson [1993] states that to assume that trans- 
fer occurs ignores research that says that performance is highly task dependent and general- 
isations from task to task are limited. Melton [1994], writing about the impact of standards 
of competence in the form of National Vocational Qualifications in British industry, states 
that; 

... it was hoped that competencies developed within one context would transfer to 
other contexts within the same occupational area. Although such transfer may happen 
readily in very simple, closely related situations, in most cases some further knowl- 
edge or skill will be needed to facilitate transfer. [p291] 

In vocational programs there must be a balance of knowledge, skill and attitude if there is 
to be maximum transfer of technology [Tedesco 1991]. The ability to recognise common 
factors and use other information to deal with a new situation will enhance the likelihood of 
transfer taking place. Oates [1992] examines the relationship between research work on 
transfer and curriculum initiatives designed to promote transfer. He sees transfer involving 
“change or adaptation” of existing skills or constructs. He writes: 

The extent of that change and the way we manage that change thus determine the 
extent and the speed with which we can leam to perform in that new task/situation. 
[ P 232] 

Constant modification of skills and strategies is needed in work and everyday life, and one 
way to assist this modification is to provide individuals with an understanding of the con- 
texts in which their skills were originally developed and of other contexts in which they 
might be subsequently used [Melton 1994]. 

ACAL [1992] states that if transfer of learning is an important outcome then training in 
transfer should be incorporated as a skill within the training. It suggests that students are 
given practice in thinking through the adaptation of ideas to different applications and that 
ability to transfer learning is identified as a required learning outcome. Mikulecky and 
Ehlinger [1986] found that instructional activities which increase metacognitive awareness 
should be incorporated into literacy programs as use of the metacognitive aspects of litera- 
cy consistently and significantly correlated with job performance. Melton [1994] cites a 
‘meta-competence’ described by Fleming. This provides individuals with a ‘critical, adapt- 
able perspective on, and ability to manipulate, one’s own competencies’, similar to the 
metacognitive awareness described by Mikulecky and Ehlinger [1986]. 

With competency based assessment, we may be claiming that individuals possess skills 
that they do not in fact possess. Watson [1994] states there is a fundamental problem in 
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making valid judgements about real-life competencies on the evidence of one successful perfor- 
mance of an assessment task in a college setting. ACAL [1992] further comments on the effect 
of attitudes and personal attributes in the demonstration of competency. As Norton [1991] 
points out there are a myriad of complex and interrelated skills involved in using literacy effec- 
tively, and competence depends on interest, attitude, prior knowledge, purpose and all those 
affective areas that are impossible to measure in CBT format. 

The literature indicates that there are many factors to consider in the transference of competence 
which are much more than just the acquisition of literacy skills. This lends weight to the teach- 
ers’ concerns that initiated this research. 

Design of the Study 

The 1 0 participants for this project, 4 female and 6 male, were Newstart clients who had 
achieved competencies working with literacy and numeracy modules in the CNL03 curriculum. 
They were all clients referred to the Bayside Campus by the CES, for full time literacy courses in 
1995 and who met the following criteria: 

• completed the course and therefore had achieved some competencies [Three others did not 
complete the course] 

• were contactable on phone numbers given on their files [Some clients had left the district 
and others could not be traced. Six clients were in this category] 

• had time to attend an interview [One person was working full time and also had a job at 
night and therefore felt unable to give time to the project]. 

Only 4 clients who had completed the course and who could be contacted were not interviewed 
because of time constraints. 

The participants were long term unemployed [usually for 1 year or more] persons who were reg- 
istered with the CES. They were initially interviewed by the CES and recommended for a litera- 
cy assessment. As a result of this one hour interview, at the Bayside Campus, the participants 
were then invited to attend a full time course of 200 hours [Newstart]. These courses run each 
term for approximately 10 weeks on 4 days each week, from 9am to 3pm. 

The participants were recommended for courses at levels 1.1 or 2.1 [beginner or developing liter- 
acy levels] and 1.2 or 2.2 [beginner or developing literacy with a vocational emphasis] on the 
LLCM [Interim Literacy Course Matrix] and were therefore enrolled in CNL03 in the appropriate 
modules from LIT101 to LIT 103, NUM101 to NUM104 and all students were enrolled in 
LIT104. Class size was approximately 10 students. 

Ages of the participants varied from 17 to 45 years and, at the time of the project, their employ- 
ment status was varied. 

• Two of the participants, one male and one female, both in their 20s, are now employed full 
time by local councils, one is on a traineeship and the other on a CES job skills program. 

• One participant gained a full time position on leaving the course, but injured his back. He 
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is now working part time whilst looking for another full time job. 

• Another participant is working as a volunteer at a local state school where he is in 
charge of a group of children one afternoon a week. He works with the children in 
the shade house propagating and repotting plants. He also assists the groundsman 
with jobs around the school and does casual work. 

• Two male participants in their early 40s [who had been employed for about 20 years] 
are still currently unemployed and looking for work. One has completed a short per- 
maculture course since leaving the Newstart program and is currently attending a part 
time literacy course. 

• Four of the participants are currently in a second Newstart program. One of these 
participants is working full time on afternoon shift and the other three have a days 
work experience each week. 

Communities of Practice 

Since literacy is significantly influenced by the contexts in which it occurs the context of 
the college and the clients’ context of use outside the college were studied. These contexts 
are described as “communities of practice” [Billet 1994b], A community of practice is a 
set of relationships that exist amongst participants within that community and the communi- 
ty’s relationship with other communities of practice. Communities of practice are distin- 
guished from each other by the activities that take place within them. Formal learning insti- 
tutions also possess a strong set of cultural practices which are associated with the goals of 
that institution and for this reason the Bayside College was studied as a community of 
practice. The norms and practices of each community were considered in order to assess 
whether transfer was ‘near’ or ‘far’ since this could influence the likelihood of skills being 
transferred. Near transfer is between situations that are similar but far transfer refers to sit- 
uations that do not have many similarities [Royer, 1979, cited in Billet 1994a]. 

The contexts of the college environment, 3 worksites and the CES were studied by inter- 
viewing the teachers involved with the Newstart program, supervisors at worksites of par- 
ticipants employed and a case manager at the CES. A supervisor at a commercial worksite 
was not available for interview as he could not spare the time in the lead up to Christmas. 
However, he did agree to a brief telephone interview. Another participant employed in pri- 
vate industry indicated that his boss would not be involved in such an interview as work 
time must be spent productively. 

Data Collection and Instrumentation 

As literacy is specific to each context in which it is found, a descriptive study [Gay 1987] 
was considered the most appropriate form of research. 

Ideally, an observational study may have been the best approach but this technique was not 
used due to the time required to establish a comfortable relationship with all the participants 
in a community of practice before meaningful data could be collected. 

Interviews were used to collect data from the participants themselves and from major participants 
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in their communities of practice. These semi-structured interviews lasted up to one hour. A tape 
recorder was not used as it was felt that this could have made participants less relaxed and less 
frank in their discussions. One researcher asked the questions and the other recorded the infor- 
mation including some verbatim quotes. The recorder also asked an occasional question. Having 
both researchers present acted as a check on bias. When both researchers could not be present 
someone else took over the role of scribe. All interviews were typed the same day which ensured 
that notes were critiqued accurately. 

Data collected at these interviews from the course participants concerned their use of the literacy 
competencies achieved on the Newstart course in other contexts, that is, at work, at home or 
when visiting the CES office. The data collected from teachers, supervisors, employers and the 
CES case manager provided information on the different communities of practice. 

Interviews with students were conducted in a small room where most of the participants had been 
interviewed prior to beginning the Newstart course. The participants sat in comfortable chairs 
alongside the researcher asking the questions. The researchers had a good relationship with the 
participants as they had worked beside them at times during the Newstart course although not in 
a teaching role. The interviews were informal and the frank responses indicated that participants 
trusted that information given was confidential. The interviewer was sensitive to participants’ 
reactions to certain questions and proceeded accordingly. 

Interviews with teachers were conducted in the college and interviews with supervisors, employ- 
ers and the CES case manager were conducted at each worksite. 

Triangulation, “the use of two of more methods of data collection” [Bums 1990 p249,] was used 
in the study to improve the validity of the data. Quantitative data were collected from partici- 
pants regarding their perception of their ability to use the competencies achieved in the Newstart 
course in other contexts. Participants rated their ability on a three point scale. The semi-struc- 
tured interviews also addressed similar information but not in the form of such overt questions. 

Where access could be gained to the worksites of those in employment, interviews with supervi- 
sors provided information about the subjects from a different viewpoint. 

Construction of the interview guide 

In order to obtain comparable data, interviews should be conducted in essentially the same man- 
ner [Gay 1 987, p203]. The interviews were semi structured, with structured questions followed 
by clarifying, unstructured questions that encouraged interviewees to elaborate on their initial 
responses. 

Two interview guides were prepared, one for the participants of the research and one for the 
teachers of the Newstart course, the supervisors at worksites and the CES case, managers. The lat- 
ter interview guide gave information about the different communities of practice. The two inter- 
view guides had certain similarities in that they both elicited information about the contexts of 
operation of participants [see Appendixes A & B]. 

Interview guide for participants 

In constructing the interview guide for the participants the validity of self-reporting was considered. 
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Self-reporting is often questioned as it has been claimed that verbal data can be founded on 
opinion rather that actual events or changes in behaviour [Billett 1994b], For this reason a 
critical incident approach was used to increase validity. Participants were asked to recall 
actual events in the workplace, home or CES office where they had used reading, writing, 
numeracy or communication skills learnt and practised on the course. We anticipated the 
recall of these incidents would provide evidence of whether those literacy/numeracy com- 
petencies which had contributed toward the achievement of competence in the learning out- 
comes were able to be used or transferred. They were asked to recall a time when “things 
went really well” , a time “when they had a problem to solve and they did something about 
it” and a time “when things did not go well”. These three questions provided an opportunity 
for participants to give evidence of their ability to transfer competence, to modify and 
adapt a skill to a new context or be unable to transfer competence. 

The participants were also asked to state how the Newstart course had prepared them, or 
not prepared them, for work or looking for work. 

They were asked to think about the settings of college and work, home or the CES with 
consideration given to the performance of tasks, relationships with peers, supervisors and 
teachers, and the responsibilities of each setting, to allow consideration of near or far trans- 
fer. 

Participants were also asked to rate on a three point scale their perceptions of their current 
competencies in the learning outcomes achieved at the end of the Newstart course. This 
included both literacy and numeracy competencies. 



Interview guide for teachers, supervisors and CES Case Manager 

In this interview guide information was sought about the different communities of practice, 
This included: location and layout, the major participants and expectations of their behav- 
iour with regard to performance of tasks, social interaction and the relationships between 
teacher/participant, employer/employee and case manager/job seeker. Information was 
sought regarding the values and concerns of each community both internally and with the 
broader community. The explicit rules of each community were also established. 

The interview guides were trialed on a student and teacher before general use. 



Results and discussion 
Analysis of participant data 

Confidence in using literacy and numeracy competencies 

The quantitative data showed that all 10 participants in the study still felt confident about 
using the majority of their stated literacy and numeracy competencies. 

The literacy competencies where some participants thought they might need help related to 
using texts from organisations/systems [L03] and interacting with the community [L04] 
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where a written response was required. For example, filling in a complicated form, the example 
given was a driver’s licence form, and writing letters where help would be needed with appropri- 
ate vocabulary, tone and spelling. One participant did not feel confident about reading some job 
seeking text. Participants responded to the question of confidence in these learning outcomes as 
“yes, with some help”. They had some knowledge of what was required but were at the assisted 
stage of competence. 

In numeracy, all participants felt confident about their stated competencies with regard to under- 
standing shape and direction, including the use of maps, most measurement and reading timeta- 
bles [those applicable to the subject’s context of use] and simple graphs. The numeracy compe- 
tencies where participants felt they might still need help related to numeration [LOl]. For exam- 
ple, understanding decimal fraction to hundredths and handling money in a real-life context. 
Another participant [7] commented on his confidence in operating with numbers to 20 as: 

It depends on how I feel. If I’m tired I’m not confident but with a clear mind it will come 
back to me. It depends on whether I’m tired or not as to how much I remember. 

In relation to the learning outcome dealing with measurement, two female participants comment- 
ed that they felt confident about the concepts of mass, capacity and time but they may need help 
with length. They stated that they used mass in food shopping, and capacity in cooking but they 
did not use length very much as they had little need to measure things accurately. 

A male participant said that he did not always feel confident about weighing things at work. He 
described it in this way, 

When I weigh things up in kilos at work, at times I feel hesitant and I think what does a 
kilo look like? Then at other times metric measurement seems OK and then I feel good 
because I feel like one of the other workers. 

This participant commented that his wife always did the shopping and he did not weigh things at 
home. In relation to these participant comments, are these gender or stereotyping issues? 

One participant who felt confident that he would be able to demonstrate competence in all six 
learning outcomes in literacy and in the four learning outcomes in numeracy said, 

If I’ve got to do it I’ll always try. If you stuff it up, so what, you’ll learn something any- 
way”. 

This indicates that confidence is not only about demonstrating a skill competently, it also 
involves being willing to accept making mistakes as part of the learning process. 

In contrast to this, a competent participant who felt confident with most of her achieved learning 
outcomes was still hesitant as to whether she would be able to demonstrate competence “out 
there”, referring to an unknown context. She commented that for letters of application she had 
been marked competent “but I didn’t believe I was competent. What do I do when I leave col- 
lege? I’d panic and scream”. 

This participant said that her teacher had given her a procedure to follow in the form of a model 
for a letter of application in response to a CES job card or job advertisement. She said, 
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I know I need to put: 

Date 

Their name 
My name 
Dear Sir/Madam 

1st paragraph - a reference to the job you are applying for 
2nd paragraph - who you are/what you can offer 
3rd paragraph - list skills 

4th paragraph - enclose resume and refer to availability for interview. 

Despite being able to give this knowledge readily she still felt she did not have the skill to 
use it in a non-college environment, in this case, at home. 

This indicates that the ability to transfer competence is not only about having the knowl- 
edge but it is about knowing how to use that knowledge and having a belief in yourself 
being able to use the knowledge in an unfamiliar context. She further said; “I want to be 
told I can do a particular skill in a real life context ”. 

One participant did comment that he had used the knowledge he had gained on the course 
about writing letters for job applications when being interviewed on the telephone. The 
participant described the result of this as reaching a collaborative agreement that the boss 
use him as a casual employee. This demonstrates an incidence of far transfer. 

In addition, students need to realise that they won’t always “get it right” immediately and 
they must be able to accept mistakes and learn from them. The participant rated her com- 
petence against the standard of being able to produce a letter of application, immediately, 
independently and perfectly. But, competence does not necessarily mean “getting the skill 
right first time”. It might mean knowing the process to go through in order to get the final 
perfect copy. She had no real appreciation of the normalcy of multiple drafts, collaboration 
and acceptance of a perhaps less than perfect, final copy due to time constraints, for exam- 
ple. 

When a literacy/numeracy event went well 

Interviews revealed that all participants could recall a time when a literacy event went well. 
Examples of events were: completing a form, answering the phone and taking messages, 
numeracy skills of addition and subtraction involving money, communication skills when 
meeting people for the first time and using a computer. 

Successfully completing a social security form was cited by one participant. She said the 
Newstart course had helped her because she had worked through forms, practised, seen, 
analysed and remembered forms. 

I’d adjusted to their language so it wasn’t confusing. When I read the form I knew 
what they wanted. Before I’d freak out. 

Three participants mentioned using communication skills as a time when things went really 
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well. One described this in terms of answering the phone and writing down messages: 

I feel comfortable doing this and feel successful at it because we practised using the phone 
on the Newstart course and we were told how to answer the phone properly. 

This is an example of ‘near’ transfer of skills as the participant was able to use the telephone and 
message writing skills practised on the course in another context. 

The two other participants commented on their awareness of using communication skills when 
meeting people. One described this in terms of a successful job interview and the other on meet- 
ing a colleague in his new workplace. Both commented on using some aspects of the Newstart 
course to help them with this although they both said that practice in other social situations had 
also assisted them with communication skills. 

The participant who cited using the computer said: 

I have used it [the computer] at home to help the children with their school work and have 
taught the youngest how to use 5.1. Mike, my husband, couldn’t do merges and I could 
show him. I thought, I know something that you don’t know. I felt great! 

This is another example of ‘near’ transfer of skills as the participant had the same computer pro- 
gram at home as she had used on the course. The participant thought she was successful 
because: 

I had a great teacher and it wasn’t as confusing as I thought it would be and I worked hard 
to understand the computer and understood it. 

She said that the teacher provided a relaxed environment, she had a good sense of humour and 
made learning fun, techniques which will affect a student’s attitude toward learning. 

Another participant recalled an occasion when he had found reading and writing a lot easier 
since leaving the course but he felt that this was not only related to an improvement in skills, but 
also to the fact that what he had to read and write about [horticulture] was a topic he was really 
interested in, “I found it meaningful and really wanted to do it”. 

When you had a problem to solve 

All participants, with the exception of one, could speak about a time when they had a problem to 
solve and were able to do something about it. Some examples given were: using spelling strate- 
gies, clarifying and confirming information, contacting a local member of parliament, keeping 
cool in a potentially explosive situation and talking things through instead of getting caught up in 
an argument. 

One participant spoke about the need for clarifying and confirming information at work in order 
to know what job had to be done. He said that he had taught his mother and father to clarify and 
confirm and recalled that the teachers on the course all talked about clarifying and confirming 
information. 

Breaking up words into syllables was a spelling strategy used by one participant when he came 
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across a word in a letter that he didn’t know. He described what he did: 

I suddenly remembered to break up words into syllables. That was wonderful because 
I didn’t have to ring my wife up at work and ask her to spell a word. 

This participant is currently working with an adult literacy tutor. He commented that on the 
course he gained the confidence to attack words using given strategies and working one to 
one with a tutor had reinforced this. 

One participant described an instance when she had the confidence to phone her local mem- 
ber regarding a problem in her street. She said that she was able to do this because the 
Newstart group had visited the local council and they had been given information about 
their local member of parliament. In addition, they had practised telephone skills on the 
Newstart course so she felt well-prepared to make the call. 

She commented that on industrial visits the teacher had taught students to introduce them- 
selves to people they would never have approached in the past. “[The teacher] pushes you 
into situations and expects 100% ’. This participant felt that this was very good preparation 
for the future world of work. 

Using communication skills to defuse potentially explosive situations was described by two 
subjects. One subject said: 

OK, yeah, this guy was doing this and that wrong, we started to talkabout it and he 
was yelling, then we sat down and talked about it and sorted it out. 

The course had taught this participant to get along with everyone. He said that all the 
teachers all the way through the course encouraged this. 

When things didn 't go well 

Not all participants could nominate an occasion when things did not go well. Three of the 
participants could not recall such an event, one of them stating that he only saw things as 
problems and then he would look for a way to solve them. He said: 

I don’t let it worry me. When you’re not worrying it’s clearer so you can make better 
judgements and see how things are. 

Some incidents nominated were: trying to complete complicated forms, a communication 
problem which arose because the participant was not aware of the protocol involved in the 
workplace and a lack of confidence in following through a job application. 

The participant who said that things did not go well when she was trying to fill in a compli- 
cated form, was also the participant who stated that completing a form at social security 
was a time when things did go really well. Regarding the latter she said that this was 
because she had had practice filling in a variety of forms, analysing them, remembering 
them and using her visual memory. In the course she had practised filling in social security 
forms every fortnight. This indicates that she had gained the confidence and skills to com- 
plete a form she was familiar with but she was not able to complete a different form, that 

135 



Adult Literacy and Numeracy: Assessing Change 



she described as complicated, and one she had not seen before. The social security form repre- 
sented a ‘near’ transfer of skills but the other form required her to adapt her knowledge and 
skills and apply herself to an unknown context. This requires confidence and a willingness to 
attempt something that looks different. 

Another participant had difficulties at work because he did not know the job situation very well 
or exactly what he should have been doing. His job was explained very quickly and he was left 
to get on with it. He said that he needed to feel more confident and should have asked for more 
information, but he had also decided that he needed time watching an expert and this was what 
he was going to do in his spare time. 

One participant handed in her resignation at work over a communication problem she had with 
her supervisor. She described this as a personality problem. She identified that she needed to 
know how to deal with the problem instead of bottling it all up and in order to do this she needed 
to know the “chain of command” or the protocol to use in that specific workplace. She com- 
mented that at college problems were resolved quickly because the teachers were sensitive to 
what was going on and things were discussed before they got to crisis point. She had found it 
different in the workplace and said that it was important to know how to deal with conflict as 
people had to work things out for themselves. She said the course had not prepared her to deal 
with conflict although there had been a focus on communicating in teams. 

Did the Newstart course prepare you for work or looking for it? 

All participants could nominate at least one area in which the course had been valuable. 

Examples of these were: numeracy skills relating to money and metric measurement, telephone 
skills, the format for job applications, filling in forms, resume writing, communication skills, 
improved reading skills, being able to use a fax and photocopier and, most frequently mentioned, 
improved self-confidence and self-esteem. 

All participants could readily give examples of using the skills they nominated which would indi- 
cate that some of the skills learnt were being transferred. For example, one participant said that 
now he would have a go at filling out forms, several participants said that they were now more 
confident using the telephone, another participant said that now he could measure in metric, 
whereas in the past he measured in feet and inches. This meant he could now go to the hardware 
store and order timber in metric measurement just like everyone else. 

Another participant confirmed that she felt confident about using the skills she had learnt when 
she commented that, without the course, she would have been very nervous starting work experi- 
ence. Because she had practised many of the skills she would be using, she thought work experi- 
ence would be interesting and not overwhelming. [This participant was enrolled in a module at 
level 2a and was more competent in literacy skills than some other members of the group.] 

The extent of a participant’s preparation for work or for seeking employment was related to the 
level of competence that they were at when exiting the course. Those participants who exited at 
level la for certain learning outcomes, whilst feeling confident to use the competencies achieved 
using the skills identified at that level, eg. simple sentences, familiar vocabulary, still had diffi- 
culties with general reading and writing tasks in the workplace which encompass a range of skill 
levels in the one text. For example, whilst participants exiting at level la could complete their 
personal details on a form [LOl] they may have difficulty completing workplace forms requiring 
more information. 
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One participant said that he was still “not too crash hot at reading job cards”” and another 
who works in a chicken processing plant, commented: 

I feel a bit more secure about reading because now I try to read everything I see but I 
don’t know what the specials are on the wall. 

Another participant said that the course was the start of something for him. He didn’t leam 
everything that he needed to know but now he knows that he wants to leam more and can 
leam more. 

Another participant found that although she had practised writing formal letters on the 
course she was not prepared for writing the letters used in her workplace. She felt she was 
not prepared for " putting words on paper". This comment supports the findings stated ear- 
lier, when participants were asked to state how confident they felt about using their 
achieved literacy and numeracy competencies. Participants felt they may need help with 
written responses [LOs 3 & 4] to ensure that they were using the appropriate vocabulary 
and tone in letters. However, no course can prepare students for all the possible contexts 
they may operate in, it can only provide them with strategies to use in different situations 
and establish an awareness of the contextual basis of literacy. 

Apart from learning skills, one participant said that he had found the course helpful in 
developing a routine. He had had to get up early to attend the course regularly and he 
realised it was "important to have a routine because 2-3 weeks after the course if you are 
not getting work you would slip back into the old habits, so I went out and got volunteer 
work and I fix people 's cars ", 

Some comments about increased self-confidence that participants felt after the course 
were: 



The course definitely gave me the self-confidence and the self-esteem to get the job at 
the ... council. 

[It has] helped me in my confidence, brought the best out in me. There’s still more 
that has to be pushed out. It has made me a better person by making me a more com- 
plete person. It has bought me over the top to the point of no return. 



The data demonstrate that transfer of skills is taking place amongst the subjects inter- 
viewed. ACAL [1992] suggests that for transference to be successful students must be 
exposed to exercises where they practise using skills in simulated contexts. In addition, 
ACAL suggests that students need to be given practice at thinking through the adaptation 
of skills to different applications and this is most effectively achieved through group discus- 
sion. For the transference of skills that is taking place are these conditions being followed 
in the Newstart course? 



Analysis of data from communities of practice - Bayside College 

Interviews with teachers revealed their ideas about conducting the Newstart course. All the 
teachers aim to help their students increase their confidence, self-esteem and acceptance of 
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themselves. They want the students to realise that it is alright to make mistakes and they can ask 
for more information. 

The numeracy teacher [1] tries to foster independence and encourages students to persist and 
“struggle” with a problem before she assists, thus simulating the workplace where students and 
supervisors comment that the workplace demands employees who can work independently and 
show initiative, [communities of practice 1, 2 & 5]. She encourages students to talk about 
numeracy and ask questions to ensure understanding. The teacher uses group work and peer 
tutoring but also ensures participation of every member of the group by having all class members 
make presentations individually. Otherwise, some students will “sit back” and let others do all 
the work. She encourages students to recognise the strengths and weaknesses in themselves and 
others. 

Teacher 2 aims to generalise skills by giving practice of the same skill in different situations thus 
modelling transfer. She commented on the importance of listening and speaking and spends time 
developing these skills. 

Teacher 2 wanted students to “feel okay about making mistakes and for them to feel they can ask 
for more information This teacher also told students that “if you want something you have to 
demonstrate that you want it and be prepared to stretch yourself. 

Both teachers 1 and 2 encourage students to be responsible for their own direction and discour- 
age teacher dependence. 

Teacher 3 runs the class as a workplace unit, although sometimes this is broken for “chalk and 
talk”. Students must arrive on time [punctuality was considered essential], behave and dress 
appropriately, consider others, use appropriate language, clean up at the end of the day and attend 
to lights and windows and so on. The class operates in a simulated work environment, in teams 
with a team leader. The teacher expects the older students to assist the younger students and pos- 
sibly accept a leadership role. She sees role models as really important. This teacher is con- 
cerned that the environment can become “too comfortable” and she imposes change. In her inter- 
view she kept coming back to the conflict between making the students feel comfortable and the 
reality of the “outside” world. As one participant commented she “pushes you into situations 
and expects 100% Another participant said of the same teacher: 

She pushed me so much I got frustrated and angry. I was writing a letter of complaint and 
she kept insisting it had to be perfect. She pushed me hard. 

It appeared that this was a new experience for this participant to be pushed to the point of feeling 
uncomfortable, but the teacher managed to get her to stick with it and the subject did receive a 
reply to her letter. 

With regard to social behaviour amongst students on Newstart courses all teachers encourage co- 
operation and support and this is related to appropriate behaviour in the workplace [Learning out- 
come 2, group communication]. Tolerance and acceptance of differences amongst the group are 
promoted along with positive talk and positive self-talk. This creates an atmosphere of everyone 
being valued and everyone being important [Learning outcome 1, personal identity]. Students 
also have to consider the needs of teachers. Teacher 3 was concerned that students were coming 
into the staffroom and demanding time from staff so guidelines had to be established for behaviour. 
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Teacher 2 described this as " an awareness of others and place, not abusing their relation- 
ship with teachers The guidelines established regarding this issue were preparation for 
understanding and following the protocol of the organisation/system in which participants 
would find themselves [Learning outcome 3, systems communication]. 

Analysis of data from other communities of practice 

What are your expectations of employees /clients? 

Supervisors at three worksites [two councils and one manufacturing company] emphasised 
the need for employees to be able to work reliably and with minimal supervision. On two 
of these worksites it was essential to be able to work effectively as part of a team. One 
supervisor said, if a person cannot do this they are out because all work is carried out in 
three-man teams ". 

Working as part of an independent team requires workers to be honest, reliable and respon- 
sible and to follow tasks through to completion which often necessitates flexibility. All 
these qualities were mentioned as desirable by supervisors in other communities of prac- 
tice. Inherent in working as an autonomous team is the need for time management skills, 
an attribute mentioned by two other supervisors. Working in a team and effective commu- 
nication skills are incorporated in learning outcome 2, group communication. 

The two council supervisors spoke of the need for employees to be able to communicate 
well with the public, because all employees are representatives of their councils and they 
must be seen as caring for the community [learning outcome 4, community]. Council 
employees also needed to know the protocols of their worksites [learning outcome 3, organ- 
isations/systems communication] in order to keep their jobs, for example, knowing the 
appropriate dress code, being punctual, “not taking too many sickies” , knowing who to 
approach when any problems arose and knowing the workplace’s health and safety prac- 
tices. Protocols of the school mentioned by the supervisor were: participant 3 being called 
Mr ... , knowing the appropriate dress code, using appropriate language and no smoking on 
school grounds. 

The skills that supervisors required varied according to each specific community of prac- 
tice. Both council supervisors commented on the need for employees to have good com- 
munication skills when interacting with the public, both on the phone and face-to-face. 

One council supervisor said that employees needed a driving licence and some knowledge 
of basic horticultural terms and tools. But beyond this, the council was willing to support 
employees in further training and did provide training in first aid, chain saw skills, literacy 
and basic computer skills. The other council supervisor was happy to assist participant 4 
with her literacy skills where her work was for public presentation and further training pro- 
grams were made available and supported by the supervisor. At the manufacturing work- 
site, everyone starting work is trained on the job. This training period could be from 2 to 3 
weeks or 8 to 12 weeks. The supervisor commented that there was also a “heavy” induc- 
tion period for fire evacuation procedures, emergency response procedures and workplace 
health and safety training, [learning outcome 5, procedural communication]. 

These comments by supervisors indicate that it is not the skills that employees possess that 
are always of prime importance but qualities in the affective domain that are always important. 
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Punctuality, honesty, reliability, responsibility, flexibility, the ability to work in a team with mini- 
mal supervision, productivity and time management skills were attributes frequently mentioned 
by supervisors as being required on worksites today. Supervisors stated that training was provid- 
ed to give employees the skills to do the job [manufacturing worksite] or to increase skills and 
this is inevitable with multiskilling being the aim of most worksites today. 

The expectations of the communities of practice are nearly all in the affective domain but assess- 
ing the affective domain in CBT is difficult. 

These comments relating to the affective domain were fully supported by the CES case manager 
who said that "attitude was more important than skills ’ . However, he did state that he felt 
numeracy skills involved with the handling of money were important as a large number of jobs 
coming through the CES are in retail. 

He explained how influential attitude was with regard to successful employment. He comment- 
ed that the wrong attitude affected communication skills and said that a major complaint of 
employers was that employees can’t see the consequences of not serving the customer well. 
Punctuality and reliability are essential for work and "employers will give up skills for these 
attributes and then possibly train workers to pick up skills they are lacking ”. 

These attributes of the affective domain are being included in the Newstart course used in this 
study as the teachers interviewed stated that they incorporated these practices into the course. 
Supervisors commented that participants 3 and 4 displayed many of these attributes. Participant 
3’s supervisor said that participant 3 demonstrated excellent communication skills, he worked 
independently, was responsible and was modelling responsibility to the children he was working 
with, and his time management skills were good. Participant 4’s supervisor commented that par- 
ticipant 4 was good at following through projects and she was considered reliable and flexible as 
she was given the council car to drive to "run around for everyone”. 

It is difficult to determine whether the ability to demonstrate these attributes is related to the 
Newstart course or whether these students came to the course with these abilities. However, 
from comments made by these participants it appears that at least some consolidation of these 
abilities took place on the course. 

Participant 3 may be using some of the techniques modelled by the Newstart teachers because his 
own experiences of school about twenty years ago were not happy ones and were nothing like the 
way he is operating with the children he is supervising in his job. He said, When I was a kid my 
Grade 1 teacher used to pick me up and shake me by the ear”. He commented that on the 
Newstart course everyone was expected to do their best and it didn’t matter if you failed because 
someone was there to help you. He described the teachers as being more personal and their atti- 
tude infectious so that they made you want to leam, knowing there was always support there. He 
describes his role with the children as, "To help them understand and leam about what they are 
doing ”. 

Participant 4 felt that one of her responsibilities on the Newstart course was to get the work done 
for her teachers and this could be linked with her ability to follow through on projects. She 
described her responsibilities in her job as, working as quickly as possible, working well with 
others and always attending work. She also commented on the realisation that there is distance at 
work between supervisor and employee that no longer exists once you step outside of work hours 
[as described by teachers 2 and 3]. 
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If the goal for competence is that it should be transferable [NRS, 1995] we need to know 
the skills which enhance performance in a wide range of settings. Oates [1992] cites the 
CASE project which identified the following key concepts as enhancing transfer. They 
include: 

• cognitive conflict, which involves developing the ability to think by means of “strug- 
gling with intellectual problems. This means creating challenges for learners which 
do not demoralise them through constant failure. 

• reflection encouraging students to think about their own thinking [metacognition]. 

• bridging, which encourages a conscious approach to applying existing strategies to 
new situations. 

• reasoning patterns which focus on certain types of reasoning such as giving probabili- 
ty values to cause and effect relationships, control of variables and finding equilibria. 

From the data given by teachers and students, the Newstart program incorporates “cognitive 
conflict”. Teacher 1 stated that she encouraged students to “struggle” with a problem 
before giving assistance and all teachers encouraged students to be independent which 
implies having to struggle with problems independently before being given help. 
Participants stated that teacher 3 created challenges for them by "pushing them into situa- 
tions and expecting 100%" and whilst one participant found this very frustrating, she did 
not fail and in the end was pleased with all the effort she had had to make. 

Teachers did not state that they encouraged students to reflect on their thinking although 
they may do this. 

Whether bridging or reasoning patterns were used consciously by teachers was not explicit- 
ly stated, although teacher 2 did say that she generalised practise of the same skill in differ- 
ent situations which implies a modelling of transfer. 



Teachers ' and participants ’ thoughts on CBT and the curriculum used on the Newstart 
course [ CNL03 ] 



The teachers interviewed question the appropriateness of competency-based tr ainin g in the 
Adult Literacy field. With literacy, there are so many factors to be considered other than 
just the acquisition of a skill. Competence depends on interest, attitude, prior knowledge 
and purpose and all those affective areas that are impossible to measure in CBT format 
[Norton, 1991]. 



Some students interviewed [particularly those identified exiting 2.1 or 3.1 ILCM courses] 
were glad to have learning outcomes clearly defined, to know the objectives and to have 
recognition for what they had learnt clearly itemised. 

I like the learning outcomes because it lets me know what I have to do. 



The learning outcomes for computers were 
towards. 



good because they set goals to work 
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For me it was important to be marked J [competent]. Being marked J gave me the confi- 
dence to believe that I could do things, so that when I went to work I believed in myself. 

Other students held the opinion that the learning outcomes didn’t really reflect what they wanted 
to learn and were an impediment rather than an aid. 

The learning outcomes are hard to understand. I just want to know what jobs I have to do .... 

For myself, I wanted spelling ... and then there were all those 6 learning outcomes so that 
wasn’t quite what I wanted. 

As previously stated, the teachers were concerned that the competencies achieved in the course 
may be short-term competencies, completed confidently in the classroom, but not transferred to 
the workplace. Teachers spoke of trying to make students confident by assessing in as realistic 
and varied situations as possible, but also commented on many students’ “teacher-dependent 
mentality” and the college’s environment as being “too comfortable”. Nearly all students spoke 
of their lack of confidence as being the greatest barrier to transferring the competencies learnt 
and gained on the course to “outside” real-life situations, although many did say that the course 
helped in building confidence. 

Participants did comment on there being a pressure to complete the learning outcomes, 

[In literacy] we had to work hard to get all of them [LOs] and then we only did things 
superficially. I could learn things to get through but the next week I’d have to learn them 
all again to pass. 

I would like to spend more time on each new thing we learn to be really sure that I under- 
stand it. 

In class everything was rushed through. Once learnt, instead of spending time on a particu- 
lar outcome we had to move onto the next. 

The pressure to achieve specified learning outcomes was commented on by vocational students in 
a survey by Billett [1994a ]. Participants said that they were more concerned with passing exams 
and completing assignments than learning what was required for the application of their skills 
outside of the learning environment. 

When students feel that they have achieved learning outcomes but have not had time to consoli- 
date these then this will affect their confidence in putting skills into practice. This is an example 
of the critical difference spoken of by ACAL [1992] between what a person can do and what a 
person will do. How is the teacher to know? Emotional and psychological impacts on perfor- 
mance are not measurable in CBT terms. There is a danger that, in accepting that literacy can be 
measured in competencies, we will be contributing to a misguided notion that becoming literate 
simply means acquiring certain skills that can therefore be named and measured. 

Conclusion 

From consideration of the quantitative results and the qualitative comments of supervisors, a case 
manager, teachers and students, a number of conclusions can be drawn about the transfer of skills 
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from the Bayside College learning environment to the students’ real-life contexts. 

• Transfer of skills to some extent did take place for all participants. Incidents given of 
successful transfer most frequently related to ‘near’ transfer of skills, eg. filling in the 
fortnightly Social Security form, using the telephone, measuring in metric, using the 
computer. Transference was high where the simulated context was not at great vari- 
ance to the classroom where skills were developed and practised and where there were 
few extraneous complications, such as an angry supervisor or a personal illness. This 
‘near’ transfer of skills seemed particularly successful in numeracy areas such as mea- 
surement. This may be because the practical, concrete activities of the classroom 
were similar to those in real-life everyday situations and little adaptation of knowl- 
edge was required. In mathematical operations [LOl] there was sometimes a difficul- 
ty of transfer because the method was not presented in a recognisable format and, 
because of language, more variables are likely. 



• Students felt confident about using the majority of their stated literacy and numeracy 
competencies but literacy competencies where participants felt they might need some 
help related to using texts from organisations/systems [L03] and interacting with the 
community [L04], These learning outcomes represent ‘far’ transfer as they are con- 
cerned with reading and writing the texts of organisations/systems and writing letters 
of appropriate tone and vocabulary for different contexts. In ‘far’ transfer situations 
where the community of practice or workplace has a different expectation or standard 
from that learnt in the classroom then transfer of competence is obviously more diffi- 
cult. 

• Transference of competence to a new context requires knowledge, skill, confidence in 
one’s ability to complete the task and a willingness to accept mistakes. The ability to 
transfer a skill, particularly in situations of ‘far’ transfer, involves the affective 
domain. It was apparent that some participants gave evidence of a greater ability to 
expect change, adapt to change, struggle through problems, employ strategies leamt to 
a variety of situations and use metacognitive skills to transfer existing competence to 
a new situation. As the CES Case Manager said, “It’s all about attitude.” 

If the goal of competence is that it should be transferable [Billett 1994a, Coates and others, 
1995], courses need to be structured to enhance performance in a wide range of settings. 
Educators need to be aware that they teach to transfer learning outcomes/competencies to 
different situations. Transference skills need to be taught alongside those competencies 
needed for the successful achievement of the learning outcome. Students must be given 
strategies to use to enhance the adaptation of skills to new contexts. The metacognitive 
skills of adaptation, reflection and resolution and the affective areas of interest, attitude and 
confidence are integral to true competence. Competence should not indicate or reflect a 
finite knowledge and skill but should be demonstrated as an enabling and empowering abil- 
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WRITING AN A CTION RESEARCH 
PROPOSAL 

Peter Mountney 



Introduction 

A research proposal is a detailed description of a proposed study designed to investigate a given 
problem. It includes justification for the research questions to be examined, a detailed presenta- 
tion of the research steps that will be followed in collecting and analysing the required data, a 
projected time schedule and a budget. Research proposals can be brief and informal such as those 
that might be submitted by a university student prior to the preparation of an assignment or they 
can be more lengthy and formal such as those which are submitted to government and other 
funding agencies. The purpose of this chapter is to outline how to prepare a simple research pro- 
posal for a study that will use an action research approach. That purpose seems straightforward 
but it does present something of a contradiction. Generally a research proposal outlines, fairly 
clearly and in advance, a series of steps that the researcher will follow in carrying out the 
research activity. Action research, on the other hand, is an evolving and iterative activity. It is 
difficult to predict in advance what the outcomes will be. Therefore an action research proposal is 
something of a compromise between the formal structure of a research proposal and the flexibili- 
ty of action research. 

In this chapter there is some description of what action research is but that description is very cir- 
cumscribed and a short bibliography of action research “primers” is included for anyone who 
wishes to read more widely on the theory and practice of action research. 

Action research is defined by Kemmis and McTaggart (1988) as: 

... a form of collective self-reflective inquiry undertaken by participants in social situations 
to improve the rationality and justice of their practices as well as their understanding of 
these practices and the situations in which these practices are carried out ... . (p. 5) 

The linking of the terms “action” and “research” highlights the essential feature of the action 
research method: trying out ideas in practice as a means of improvement and as a means of 
increasing knowledge about the curriculum, teaching and learning. The result is improvement in 
what happens in the learning environment, and better articulation and justification of the educa- 
tional rationale for what goes on. Action research is usually concerned with a local problem and 
is normally conducted in a local setting. Its primary concern is not whether the results are gener- 
alisable to other settings, although this may be desirable, and it is not characterised by the same 
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kinds of controls usually evident in other categories of research. Whether the research is 
conducted in one classroom or in many classrooms, the teacher is very much a part of the 
process. Therefore the more research training action researchers are exposed to, the more 
likely it will be that the research will produce valid, if not generalisable, results. 

Origins of action research 

Action research has become increasingly popular as a form of inquiry over the last twenty 
or so years, not only in education but in a range of disciplines. It is generally acknowledged 
that Kurt Lewin originated the idea of conducting action research in the 1930’s (Adelman 
1993). It was used in contexts as diverse as integrated housing, equal employment opportu- 
nity , prejudice in children, the socialisation of gangs, and training for youth leaders. 
Lawrence Stenhouse’s Humanities Curriculum Project at the University of East Anglia in 
Norwich, England in the 1970’s is closely associated with the “teachers as researchers” 
movement and Stephen Kemmis was among the early action researchers at the Centre for 
Applied Research in Education (CARE) in Norwich. Kemmis returned to Australia to 
develop the idea further at Deakin University and linked into the two main sources con- 
tributing to the growth of action research in this country: the growth of school-based cur- 
riculum development and a growing professional awareness among teachers who were 
seeking new ways of working and understanding their work. 

Action research and practice 

Kemmis and McTaggart (1981) observe that to do action requires the following activities: 

• to develop apian of action to improve what is already happening, 

• to act to implement the plan, 

• to observe the effects of action in the context in which it occurs, and 

• to reflect on these effects as a basis for further planning, subsequent action and so on, 
through a succession of cycles, (p. 7) 

Zuber-Skerritt (1991) notes that the plan includes problem analysis and a strategic plan; 
action refers to the implementation of the strategic plan; observation includes an evaluation 
of the action by appropriate methods and techniques; and reflection means reflecting on the 
results of the evaluation and on the whole action and research process. The cyclical nature 
of the activities demonstrates that action research is not linear. Rather it builds on the issues 
that emerged in the preceding cycle(s) as part of a process leading to further inquiry. 

Kember and Kelly (1993) prefer to add a fifth process that they identify as ‘initial reflec- 
tion’ to the plan-act-observe-reflect cycle. Quoting Kemmis and McTaggart (1988) they 
state: 



Planning usually starts with something like a general idea. For one reason or another 
it seems desirable to reach a certain objective. Exactly how to circumscribe this objec- 
tive, and how to reach it is frequently not clear. ... If this first period of planning is 
successful, two items emerge: namely, an ‘overall plan’ of how to reach the objective 
and secondly, a decision in regard to the first step of action (p. 5). 

This action research cycle reflects a commonsense view of practice that is easily recognised 
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by experienced teachers. Teachers regularly engage in a process of planning, acting on their plans 
and reviewing and evaluating their actions for subsequent modification of the plans. Therefore a 
research process embedded in this cycle makes more sense for teachers than one that uses con- 
cepts of null hypotheses, experimental groups, dependent variables and so on. 

Action research is participatory. That is, it is research through which people work towards the 
improvement of their own practice. It is collaborative. That is, it involves those responsible for 
action in improving it, widening the collaborating group from those most directly involved to as 
man y as possible of those affected by the practices concerned. 

Action research does offer a link between theory and practice. It offers an alternative to the view 
that theory is the province of academics and, at best, is marginally useful for those engaged in the 
realities of practice. Action research involves practitioners in theorising about their practice 
through reflection on action. Existing theory can be used to understand and explain practice but, 
used in this way, it is not divorced from the realities of practice. 

Finally action research is self-research. Action researchers investigate their own practice and not 
that of others. Thus it is the academic whose role in action research is problematic. Elliot writes: 

This does not mean that the movement necessarily excludes researchers from the higher 
education sector. But it insists that the value of their contribution lies in the extent to which 
they support the practical judgement of teachers, and increases their capacity to reflect sys- 
tematically upon the complex situations they confront. The identification of problems for 
investigation, the methods of research employed, and the presentation of “findings” will all 
be executed through negotiation and collaboration with teachers. However, the future pros- 
perity of the movement will increasingly depend on teachers grasping opportunities to con- 
duct their own enquiries without external support (1981, p. 63). 

Prideaux (1995, p.12) offers a set of principles underpinning action research that practitioners can 
adopt (or adapt) as ground rules for collaborative research studies. He notes: 



1. All members are participants (no action research, no admittance). 

2. Everyone is an expert. There are no privileged experts (hence no appeals to authority and, 
conversely, no claiming to be just a practitioner and not being involved in critique). 

3. All members are equal in status and this is reflected in member-to-member communication. 

4. All members are supportive of each other and this is reflected in member-to-member com- 



5. All members take a critical perspective and this is reflected in member-to-member commu- 



6. The focus will be beyond that of individual and particular concerns. 



A distinctive feature of action research is that those affected by the planned changes have the pri- 
mary responsibility for deciding on courses of action that seem likely to lead to improvement, 
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and for evaluating the results of strategies tried out in practice. 
Action research studies - examples 
Diary-keeping and teacher development 



Reflect 



In order to record and reflect on their classroom practice, three literacy teachers initi- 
ated a study whereby they each diarised aspects of their classroom teaching activities. 
A diary is a very simple tool for gathering information which can be used to generate 
questions and to reflect on practice. It is non-intrusive because it does not require the 
assistance of an outside observer in the classroom. 

Plan 

Each teacher agreed to keep diaries for two of their classes per week for one semester. 
Act 

In compiling their diaries, the teachers combined a narration of classroom activities 
along with their reflection of those activities. What worked? What didn’t work? What 
did the students respond to actively? Why? How could it be better next time? 
Individual diary-keeping was coupled with written responses and group discussion. 
Each teacher read the diaries of the other teachers and they wrote brief responses to 
each others’ entries before their discussion period. Discussions were audio-taped and 
transcribed. 

Observe 

At the end of the semester, the diary entries, written responses and transcripts were 
analysed to determine how the three information sources interacted and what issues 
occurred most frequently. 



Diary-keeping was seen as a valuable activity in terms of describing aspects of teach- 
ing practice but at the same time the discipline of keeping a diary was time consum- 
ing. However the activity enabled the teachers to focus on improving their classroom 
practice and to gain a perspective on other teachers’ experiences. 

Self study booklets 

Reflect 

A program in Workplace Health and Safety has been exploring an alternative to up- 
front classroom delivery by a presenter. The focus of the alternative approach has 
been the use of self-study booklets. It was felt that the use of self-study booklets 
would provide a more flexible delivery strategy that would be more acceptable to 
trainees. 

Plan 

Self-study booklets were prepared for one of the two modules of the Workplace 
Health and Safety program. 



Reflect 
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Act 

Trainees attended one lecture a week for six consecutive weeks and covered the content of 
the first module of the program. For the second module, trainees were asked to read and 
complete the activities in the prepared self-study booklets. The material was self-paced but 
a time limit of six weeks was set for completion of the second module’s work. 

Observe 

A written examination was a component of both teaching approaches. A questionnaire was 
used to evaluate the trainee’s attitude to the alternative teaching and learning formats. 

Reflect 

The results were inconclusive. The examination results from the two approaches to teaching 
and learning showed no significant differences and the reactions of the trainees were mixed. 
Some favoured the flexibility of the self-study method; others favoured the classroom pre- 
sentation method, and particularly for the discussion of the case studies. 

Plan 

Plans are in place to rewrite the case study components of the self-study booklets in order 
to eliminate the identified deficiencies and to incorporate suggestions made by the trainees 
from the first study. The study will be repeated with a new group of trainees. 

(Cases adapted from Kember & Kelly, 1993.) 

These are two simple examples of action research studies. Where possible or where appropriate, 
action research studies go through several cycles or spirals of the basic phases of the study. In the 
second example above, for example, aspects of the self-study booklets are being re-written for 
the next program offering and, based on further feedback, could be re-written yet again for subse- 
quent offerings. 

The phases of the studies as described above appear to be relatively straightforward but what 
occurs in reality is less clear-cut. Pl annin g is generally not perfect and compromise and revision 
is often required. However, it is useful to describe each phase simply and separately in order to 
outline the action research process. 

Writing the action research proposal 

When writing a research proposal there are a number of issues that need to be considered and 
then addressed within the proposal. Firstly, there is the need to consider the purpose in writing 
the proposal which, to some extent, is different from the purpose in undertaking the research 
study itself. In writing a proposal, the researcher is demonstrating a capacity to design a research 
study that can be carried out successfully. In addition the researcher is “selling” the proposal. 

This “selling” might be to management for permission to undertake the study, to potential partici- 
pants or to a funding agency. Usually there are more good ideas than funds available and choices 
have to be made. A well-drafted research proposal clearly has an advantage in attracting the 
reviewer’s attention. 

It is important for the researcher to show clearly that the proposal has been fully planned, that all 
the relevant issues have been considered and that the study has been adequately costed in terms 
of time and money. 
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Another issue to be considered in writing a research proposal is that no two studies will be 
identical and, therefore, no two proposals can expect to be identical. This is especially so 
with action research which is particularly tied to context. Each study will be designed to 
satisfy its own needs. Although the following discussion is based on generic components 
for research proposals, any proposal should be individualised to its particular context. Not 
all of the components may be necessary. There may be other components specific to a par- 
ticular context that should be included. What to include and what to exclude are decisions 
that need to be made by the writer of the proposal. 

In planning a research study and in preparing a research proposal, the researcher must 
ensure that the proposal represents a project that is “doable” and, particularly, within the 
constraints that will apply - money, time, available assistance, access to participants, access 
to sites and so on. It is important for the researcher to work with tools that can be readily 
accessed to achieve useful information. A small amount of valid data is much more useful 
than a scattering of unreliable, unrelated information. 

Components of a research proposal 

At its simplest level a research proposal aims to provide information to the person who will 
approve the research study on six basic questions: 



Clearly the questions are all interrelated and often the “what” and “how” are dictated by the 
“how long”, “who for” and “how much”. Table 1 shows a summary of suggested compo- 
nents of a research proposal and how they address the six basic questions. 

Table 1 

Components of a research proposal 

Introduction (What and why) 

Statement of the research problem or issue 
Literature review 
Definition of terms 
Objectives or research questions 

Perceived audience and outcomes (Who for, who about) 

Method (How) 

Brief overview of approach 

Data sources (population and sampling) 

Data types and collection procedures 
Issues (eg. ethical) and validation procedures 
Anticipated data analysis 
Report format and content 

Human and physical resources (How long and how much) 



What? 

Why? 



Who for and who about? 
How? 

How long? 

How much? 
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The introduction 

Generally speaking the statement of the purpose of a research study should be clear and succinct. 
If the reviewer cannot identify the purpose, it is impossible to determine if the methodology is 
appropriate. The statement should convince the reviewer of the proposal’s value. It is also 
important to draw a ring around the problem so that it is clear what is included and what is left 
out. However, in an action research proposal some flexibility with regard to this requirement 
might be needed. Kemmis and McTaggart observe that: 

You do not have to begin with a “problem”. All you need is a general idea that something 
might be improved. Your general idea may stem from a promising new idea or the recogni- 
tion that existing practice falls short of aspirations ( 1981 , p. 18 ). 

If the researcher is working independently this advice is fine but, if the study is to be approved 
and/or funded by other agencies, the “general idea” will usually need to be teased out to some 
extent. Key questions the researcher might address are: 

How important is the issue to the researcher? 

How important is it for the researcher’s students? 

What opportunities are there to explore the area? 

Who might be interested in helping? 

What are the constraints - practical and political? 

How manageable is the task? 

While generality beyond the local scene is not necessarily a characteristic of action research, it is 
a desirable feature of educational research and any anticipated generality should be described. 
The two action research examples cited previously were both specific to the participants/program 
in question but both are generalisable to some extent to other contexts. Any limitations to the 
research study should be clearly identified. 

A brief review of the literature is recommended as a component of the research proposal. It will 
help to place the research study in context. No matter what the problem is, someone, somewhere, 
will have written something about it. By undertaking a brief review of the literature and related 
research, other perspectives, other issues or an appropriate methodology might be identified. Any 
terms that have a particular meaning and which might be unfamiliar to the reviewer should be 
defined. 

The objectives or research questions should flow logically from the statement of the research 
problem. The objectives should be specific, achievable and operational and they should be con- 
sistent with the methodology. 

The perceived audience and outcomes 

In developing the proposal it is important to decide for whom the research study is being under- 
taken and what are the anticipated outcomes or products of the activity. There may be several 
stakeholders, for example, literacy teachers at a specific site, literacy teachers across the state, 
administrators and so on. There may be specific products developed as a result of the research 
activity, for example, manuals or diagnostic tools etc. How will they be disseminated? How will 
the benefits of the research be shared? These are important questions that need to be addressed. 
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Method 

In describing the methodology there are a number of factors that need to be considered. 
Firstly, has the group which is the focus of the study been adequately described? Secondly, 
has a research technique that best fits the problem been chosen? The approach chosen will 
be dependent on the purpose of the research and to some extent the context. For example, is 
the study to be formative in nature such as the redesign of a more effective program? In 
this case an approach that will assist in identifying the processes needing improvement 
would be chosen. On the other hand, a study may be undertaken in a more budget-focused 
context looking to use the available funding more effectively. This would entail a more 
summative approach focusing on outcomes per dollar spent. 

Action research is based on documented evidence and data will be collected from a number 
of sources. Any of the following are appropriate sources of data depending on the particular 
research study being undertaken: 

Anecdotal records and field notes 

Document analysis 

Diaries and log books 

Portfolios 

Questionnaires 

Interviews 

Sociometric methods 
Interaction schedules and checklists 
Tapes, videos and photographs 
Tests of learner performance. 

Each method has particular strengths and weaknesses and the researchers will have to 
decide which method or methods are most appropriate for the particular study in question. 
Multiple sources of data are preferable to single sources. In the first action research exam- 
ple given above, the sources of data included the diaries, the written comments and the 
transcribed discussion. These three sources offer a kind of triangulation process. This 
process of triangulation (much like a doctor calling for second and third opinions) attempts 
to confirm validity by comparing results obtained from different perspectives, using differ- 
ent data-gathering procedures. 

Where data collection procedures might cause controversy, it is important that adequate 
consideration be given in advance to any necessary special provisions that might have to be 
put in place. For example have proper negotiations with those involved in the study been 
undertaken? Are procedures in place that will allow participants to negotiate with the 
researcher any written descriptions of their work before it is released? Where necessary, 
have explicit written authorisations been obtained? 

Is there a realistic work plan? Is the sequence of steps clear and is there sufficient time 
allowed for each step? Preparation of a time line is a good indicator of whether the research 
study is “doable” and particularly in those cases where time constraints will operate. For 
example, allowing one week for preparing, trialing and administering a questionnaire is 
clearly unrealistic. Allowing one month for a small questionnaire to a limited number of 
people would be more appropriate. A complex questionnaire to a large number of people 
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could take at least two months. The subsequent analysis and writing up of the data provided from 
the questionnaires requires a separate time consideration. Using a planning tool such as a Gantt 
chart or a PERT diagram can be a valuable component of a research proposal. 

Human and physical resources 

An action research study may be a small undertaking based in the classroom of an individual 
teacher and, as such, it would not be necessary to describe research experience or specialist quali- 
fications in any detail. On the other hand there may be more complex research studies proposed 
which would involve specialist expertise. Clearly, in such cases it would be desirable to include 
in the research proposal brief statements of pertinent experience and qualifications of the 
researchers to be involved in the study. 

In terms of physical facilities, does the proposal describe any special facilities and equipment that 
might be necessary? Are they available to the researcher when they will be required? If neces- 
sary, is there evidence of the co-operation of other agencies that might be involved. 

Budget 

Has an appropriate operational budget been outlined? The following is a fairly comprehensive 
list of the costs that you should consider when preparing your budget, although not all will be 
applicable in all situations. For example, the three teachers undertaking the diary-keeping study 
described above may not have salary costs associated with their study. In fact, their costs might 
be minimal and limited, say, to secretarial assistance for transcription of tapes and associated typ- 
ing. However, typical budget items can include: 

Salaries 

Consultant’s fees if not included in salaries 
Costs of materials, eg. questionnaires, tests 
Travel, accommodation expenses 
Office supplies and overheads 
Paper, postage, telephone, facsimile 
Computing costs: data analysis, word processing 
Printing costs. 

It is important that realistic costs be determined. Too frequently the costs associated with salaries 
and data collection are underestimated because the estimates are based on tight timelines. There 
are a variety of adages on the theme of “what can go wrong will go wrong” and a prudent 
researcher should make some provision for unexpected timelags. Program participants do not 
necessarily have the same degree of commitment to the research study as do the researchers; they 
do not share the urgency; they have other priorities. People go on holidays; questionnaires get 
lost in in-trays; approvals aren’t signed and returned as quickly as the researchers had planned. 
And so on. 

Conclusion 

The degree of detail needed in a research proposal is determined by the purpose for which it is 
being written. A research proposal prepared by a group of teachers for use in their own class- 
rooms can be a simple document. If the proposal is to be submitted to management for authorisation 
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to undertake the study, a more detailed document might be required to obtain permission 
and to justify the expenditure on the necessary human and physical resources. A research 
proposal being submitted to an outside funding agency and particularly if the proposal is in 
competition with other research studies for funding will need to be much more comprehen- 
sively described. Not only must it be done competently, it must also “grab” the attention of 
the reviewers. It doesn’t have to be a big project but it should be innovative. 

Peter Mountney 
Griffith University 
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